diff --git a/.env.example b/.env.example index a513fca..a12c173 100644 --- a/.env.example +++ b/.env.example @@ -1,4 +1,7 @@ -SALESFORCE_USERNAME=username@domain.com -SALESFORCE_PASSWORD=password -SALESFORCE_SECURITY_TOKEN= -SALESFORCE_DOMAIN=login / company.my \ No newline at end of file +# Salesforce credentials +SALESFORCE_USERNAME=you@example.com +SALESFORCE_PASSWORD=your_password +# Security token (required unless your IP is whitelisted) +SALESFORCE_SECURITY_TOKEN=your_security_token +# Domain: "login" (production), "test" (sandbox), or your My Domain subdomain (e.g. "acme") +SALESFORCE_DOMAIN=login diff --git a/README.md b/README.md index 822b918..0eff71b 100644 --- a/README.md +++ b/README.md @@ -1,219 +1,61 @@ # Salesforce Python Template -A robust Python template designed for seamless interaction with Salesforce. This project enables bulk data operations, SOQL queries, CSV processing, interactive field mapping, and data enrichment via web scraping. +A concise template for Salesforce data workflows in Python. It includes: +- CSV upload via the SObject Collections endpoint (batched up to 200 records). +- SOQL query to CSV. +- An enrichment demo that scrapes public pages and updates records after confirmation. -## Features - -- **Salesforce Authentication:** Utilises environment variables for secure access -- **Bulk Data Operations:** Supports insert, update, delete, and upsert operations -- **SOQL Query Execution:** Run queries against Salesforce and export results to CSV -- **Interactive Field Mapping:** Map CSV columns to Salesforce fields during upload -- **Data Enrichment:** Enhances Salesforce records by scraping additional company information -- **Customisable Configuration:** Easily adjust settings via a YAML configuration file -- **Comprehensive Logging and Error Handling** -- **Unit and Integration Tests:** Ensuring high code quality with pytest - -## Project Setup - -### 1. Clone the Repository +## Quickstart ```bash -git clone -cd Salesforce-Python-Template -``` - -### 2. Create and Activate a Virtual Environment -```bash -python3 -m venv venv +python -m venv venv source venv/bin/activate - -# Confirm Python version (3.9 or higher required) -python --version -``` - -### 3. Install Dependencies -```bash pip install -r requirements.txt -``` -### 4. Set Up Environment Variables -Create a copy of the environment file and update it with your Salesforce credentials: - -```bash cp .env.example .env -``` +# edit .env with your Salesforce credentials -Open the `.env` file in your preferred editor and configure the following variables: -```plaintext -SALESFORCE_USERNAME=your_username -SALESFORCE_PASSWORD=your_password -SALESFORCE_SECURITY_TOKEN=your_token -SALESFORCE_DOMAIN=login # Use 'test' for sandbox environments -``` - -## Configuration - -Review and modify the configuration file located at `config/config.yaml` to suit your requirements. - -### Key Configuration Areas: -- **Salesforce API Settings:** API version, batch size, and timeout -- **Logging:** Log level, format, and file destination -- **CSV Processing:** Encoding, delimiter, and directories for input and error files -- **Data Enrichment Fields:** Specify which fields to update for different Salesforce objects - -### Example Configuration: -```yaml -api: - version: '57.0' - batch_size: 200 - timeout: 30 - -logging: - level: INFO - format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s' - file: 'errors/app.log' - -csv: - encoding: 'utf-8' - delimiter: ',' - error_directory: 'errors' - input_directory: 'input' - -enrichment: - fields: - Account: - - Phone - - Website - - BillingStreet - - BillingCity - - BillingState - - BillingPostalCode - - BillingCountry - Contact: - - Phone - - Email - - MailingStreet - - MailingCity - - MailingState - - MailingPostalCode - - MailingCountry - Lead: - - Phone - - Email - - Street - - City - - State - - PostalCode - - Country +python scripts/query_data.py "SELECT Id, Name FROM Account" output/accounts.csv ``` -## Usage - -### 1. Upload Data Script (`scripts/upload_data.py`) - -This script uploads CSV data to Salesforce with interactive field mapping. +## Configuration overview +- `.env` (see `.env.example`) + - `SALESFORCE_USERNAME`, `SALESFORCE_PASSWORD`, `SALESFORCE_SECURITY_TOKEN` + - `SALESFORCE_DOMAIN` is `login` (prod), `test` (sandbox), or your My Domain subdomain +- `config/config.yaml` + - `api`: API version, default batch size, timeout + - `logging`: log level and log file (default `errors/app.log`) + - `csv`: encoding/delimiter and error/input directories + - `enrichment`: default fields per object for `scripts/manipulate_data.py` -#### Supported Operations: -- insert -- update -- delete -- upsert +## Scripts +- `scripts/upload_data.py` + - Uses SObject Collections (composite/sobjects), not the Bulk API. + - Interactive field mapping; insert/update/delete/upsert. + - Upsert requires `--external_id_field`. + - Writes per-record results to `results/` (success and error CSVs). +- `scripts/query_data.py`: runs SOQL and writes a CSV. +- `scripts/manipulate_data.py`: looks up a record, scrapes public pages, and proposes updates for confirmation. -#### Features: -- **Field Mapping:** Interactive CSV headers to Salesforce fields mapping -- **Batch Processing:** Records processed in configurable batches (up to 200 records per call) -- **Flexible Operations:** Support for all major bulk operations - -#### Command-Line Options: -| Option | Description | Required | -|--------|-------------|----------| -| `csv_file` | Path to the CSV file | Yes | -| `object_name` | Salesforce object name (e.g., Account, Contact) | Yes | -| `--operation` | Operation type (insert, update, delete, upsert) | No (default: insert) | -| `--config` | Path to configuration file | No (default: config/config.yaml) | -| `--external_id_field` | External ID field for upsert | Only for upsert | -| `--batch_size` | Number of records per API call | No | - -#### Example Usage: +### Usage examples ```bash -source venv/bin/activate python scripts/upload_data.py path/to/input.csv Account --operation insert -``` - -### 2. Query Data Script (`scripts/query_data.py`) - -Execute SOQL queries and export results to CSV. - -#### Command-Line Options: -| Option | Description | Required | -|--------|-------------|----------| -| `soql` | SOQL query (in quotes) | Yes | -| `output_file` | CSV output file path | Yes | -| `--config` | Path to configuration file | No | - -#### Example Usage: -```bash -source venv/bin/activate +python scripts/upload_data.py path/to/input.csv Case --operation upsert --external_id_field CaseNumber python scripts/query_data.py "SELECT Id, Name FROM Account" output/accounts.csv +python scripts/manipulate_data.py 0016g000009yzfpAAA Account --fields Phone Website ``` -### 3. Data Enrichment Script (`scripts/manipulate_data.py`) - -Enrich Salesforce records with additional data through web scraping. - -#### Command-Line Options: -| Option | Description | Required | -|--------|-------------|----------| -| `record_id` | Salesforce Record ID | Yes | -| `sobject_type` | Object type (Account, Contact, Lead) | Yes | -| `--config` | Path to configuration file | No | -| `--fields` | Specific fields to update | No | - -#### Example Usage: -```bash -source venv/bin/activate -python scripts/manipulate_data.py 0016g000009yzfpAAA Account --fields Phone Website BillingStreet -``` +## Salesforce client helpers +`src/salesforce.py` includes Bulk API helpers (`bulk_insert`, `bulk_update`) and a `query` helper. The upload script does not use these bulk helpers; it batches records through the SObject Collections REST endpoint. ## Testing +- Unit tests: `tests/test_salesforce.py`, `tests/test_bulkification.py`, `tests/test_manipulate_data.py` (with mocked HTTP). +- Integration tests: `tests/test_upload_operations.py` runs live against Salesforce and creates/deletes Case records. `tests/test_manipulate_data.py::test_full_integration` is marked integration but skipped by default. -Run the complete test suite: +Run all tests (requires Salesforce credentials and will create/delete Case records): ```bash -pytest tests/ -``` - -Generate test coverage report: -```bash -pytest --cov=src tests/ +pytest ``` ## Troubleshooting - -### Common Issues and Solutions: - -1. **ModuleNotFoundError for 'src'** - - Run scripts from project root - - Adjust PYTHONPATH if needed - -2. **Authentication Failures** - - Verify Salesforce credentials in `.env` - - Confirm API access is enabled - -3. **API Errors** - - Check API version in `config/config.yaml` - - Verify user permissions in Salesforce - -## Contributing - -We welcome contributions! Please follow these steps: - -1. Fork the repository -2. Create a feature branch -3. Submit a pull request with detailed descriptions - -## License - -This project is licensed under the MIT License. - ---- - -Thank you for using the Salesforce Python Template. We hope this tool streamlines your Salesforce data operations and enrichments. For any issues or further assistance, please open an issue on GitHub. \ No newline at end of file +- `ModuleNotFoundError: src`: run scripts from the project root. +- Authentication errors: confirm `.env` values, domain (`login`/`test`/My Domain), and API access.