Skip to content

Commit 91e6053

Browse files
lexxailexxai
committed
Split README.md
1 parent b69c2f4 commit 91e6053

10 files changed

+1166
-544
lines changed

‎README-long.md‎

Lines changed: 584 additions & 0 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 9 additions & 544 deletions
Large diffs are not rendered by default.

‎docs/part1-introduction.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Proxmox Cluster Tasks
2+
3+
## Purpose of the Project
4+
5+
The **Proxmox Cluster Tasks** repository is designed to simplify and automate daily administrative tasks within a Proxmox Virtual Environment (VE) cluster. The project focuses on enhancing the efficiency of cluster management by providing tools and wrappers for interacting with Proxmox through various backends and APIs.
6+
7+
### Key Features
8+
9+
#### Daily Task Automation
10+
- **Clone VMs from Templates**: Easily create virtual machines based on predefined templates.
11+
- **Distribute Cloned VMs Across Nodes**: Automatically balance VM distribution across cluster nodes.
12+
- **Configure VM Settings**:
13+
- **Replication**: Set up replication policies for high availability.
14+
- **Backup**: Automate backup configurations.
15+
- **Networking**: Configure network interfaces and firewall rules.
16+
- **High Availability (HA) Settings**:
17+
- Define HA groups and priorities for VMs.
18+
- For example, deploy `instance_02` on node `c02`, configure replication to nodes `c01` and `c03`, and assign HA settings prioritizing `c02`.
19+
20+
#### Multi-Backend Support
21+
- **HTTPS**: Direct API interaction using an HTTPS wrapper.
22+
- **CLI**: Programmatic execution of Proxmox CLI commands.
23+
- **SSH**: Manage clusters securely over SSH.
24+
25+
#### Flexible Execution Modes
26+
- **Asynchronous Mode**: Perform non-blocking, concurrent operations for time-sensitive tasks.
27+
- **Synchronous Mode**: Sequential execution for straightforward tasks.
28+
29+
---
30+
31+
The primary goal of this project is to streamline Proxmox VE cluster management, reduce manual effort, and improve operational consistency.

‎docs/part2-api-reference.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
## API Reference
2+
- [Proxmox VE API Documentation](https://pve.proxmox.com/wiki/Proxmox_VE_API)
3+
- [Proxmox VE API Viewer](https://pve.proxmox.com/pve-docs/api-viewer)

‎docs/part3-setup.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
## Setup
2+
3+
### Backends
4+
Supported backends:
5+
```python
6+
BACKENDS_NAMES = ["https", "cli", "ssh"]
7+
```
8+
9+
#### Automatically Register Backends
10+
To use backends, you can register all available backends or a specific subset using a configuration file with `register_backends()`:
11+
12+
```python
13+
# Register all backends
14+
register_backends()
15+
16+
# Register a single backend
17+
register_backends("https")
18+
19+
# Register specific backends
20+
register_backends(["https", "ssh"])
21+
```
22+
23+
Backends are available for both synchronous and asynchronous code.
24+
25+
#### Manually Create an Instance of a Backend
26+
Alternatively, you can manually create a backend instance by importing the corresponding module:
27+
28+
```python
29+
from ext_api.backends.backend_https import ProxmoxHTTPSBackend
30+
31+
backend = ProxmoxHTTPSBackend(
32+
base_url="<BASE_URL>",
33+
entry_point="<ENTRY_POINT>",
34+
token="<TOKEN>",
35+
verify_ssl=True
36+
)
37+
```

‎docs/part4-1-proxmoxapi-examples.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
2+
### ProxmoxAPI Class. Examples.
3+
The `ProxmoxAPI` class provides a context-friendly interface to interact with Proxmox. It supports using both manually created backend instances or automatically created instances via `backend_name` and `backend_type` parameters. Defaults: `backend_name="https"`, `backend_type="sync"`.
4+
5+
#### Example: Registered Backends (`https` in `sync` Mode)
6+
```python
7+
from ext_api.backends.registry import register_backends
8+
from ext_api.proxmox_api import ProxmoxAPI
9+
10+
register_backends("https")
11+
ext_api = ProxmoxAPI(backend_name="https", backend_type="sync")
12+
with ext_api as api:
13+
print(api.version.get(filter_keys="version"))
14+
nodes = api.nodes.get(filter_keys="node")
15+
node = nodes[0]
16+
print(api.cluster.ha.groups.get())
17+
print(api.nodes(node).status.get(filter_keys=["kversion", "uptime"]))
18+
print(api.nodes(node).status.get(filter_keys="current-kernel.release"))
19+
api.cluster.ha.groups.create(data={"group": "test_group_name", "nodes": ",".join(nodes[:3])})
20+
print(api.cluster.ha.groups("test_group_name").get(filter_keys=["group", "nodes"]))
21+
api.cluster.ha.groups("test_group_name").delete()
22+
```
23+
24+
#### Example: Registered Backends (`https` in `async` Mode)
25+
```python
26+
from ext_api.backends.registry import register_backends
27+
from ext_api.proxmox_api import ProxmoxAPI
28+
29+
async def async_main():
30+
register_backends("https")
31+
ext_api = ProxmoxAPI(backend_name="https", backend_type="async")
32+
async with ext_api as api:
33+
print(await api.version.get(filter_keys="version"))
34+
print(await api.cluster.ha.groups.get())
35+
36+
asyncio.run(async_main())
37+
```
38+
39+
#### Example: Registered Backends (`ssh` in `sync` Mode)
40+
```python
41+
from ext_api.backends.registry import register_backends
42+
from ext_api.proxmox_api import ProxmoxAPI
43+
44+
register_backends("ssh")
45+
ext_api = ProxmoxAPI(backend_name="ssh", backend_type="sync")
46+
with ext_api as api:
47+
print(api.version.get(filter_keys="version"))
48+
```
49+
50+
#### Example: Manually Created Backend (`https` in `sync` Mode)
51+
```python
52+
from ext_api.backends.backend_https import ProxmoxHTTPSBackend
53+
from ext_api.proxmox_api import ProxmoxAPI
54+
55+
backend = ProxmoxHTTPSBackend(
56+
base_url="https://proxmox.local:8006",
57+
token="user@pam!user_api=XXXX-YYYY-.....",
58+
verify_ssl=False
59+
)
60+
61+
ext_api = ProxmoxAPI(backend=backend, backend_name="https", backend_type="sync")
62+
with ext_api as api:
63+
print(api.version.get(filter_keys="version"))
64+
```
65+
66+
These examples provide flexibility for API usage, allowing you to control request preparation and execution explicitly.
67+

‎docs/part4-2-advanced-examples.md‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
### ProxmoxAPI Class. Examples. Advanced Concurrent Usage and Low-Level API Requests
2+
3+
4+
#### Example: Multiple Parallel requests with same API Instance (https in Async Mode)
5+
In this example, one API instances with reusing the same backend session are created and used in parallel for asynchronous operations:
6+
```python
7+
async def async_main():
8+
register_backends("https")
9+
ext_api = ProxmoxAPI(backend_name="https", backend_type="async")
10+
async with ext_api as api:
11+
tasks = []
12+
for _ in range(8):
13+
logger.info(len(tasks))
14+
tasks.append(api.version.get(filter_keys="version"))
15+
logger.info("Waiting for results... of resources: %s", len(tasks))
16+
results = await asyncio.gather(*tasks)
17+
logger.info(results)
18+
```
19+
Results:
20+
```log
21+
DEBUG: Creating backend: https of type: async
22+
INFO: 0
23+
DEBUG: NEW task_id: 4be8ee71-218d-43ff-b62a-6fb15827bfe0
24+
INFO: 1
25+
DEBUG: NEW task_id: a55c5863-5634-4a2f-a379-9eb73046caa0
26+
INFO: 2
27+
DEBUG: NEW task_id: 8399e980-a6d6-4225-902f-3bcd7a504b7c
28+
INFO: 3
29+
DEBUG: NEW task_id: eea05041-1415-43ab-8c79-c1f48daed260
30+
INFO: 4
31+
DEBUG: NEW task_id: 42add8a3-e2c4-4206-ac83-ba694c3b9ed2
32+
INFO: 5
33+
DEBUG: NEW task_id: 4fd30c33-67c2-42f0-8150-67e27ca5ac45
34+
INFO: 6
35+
DEBUG: NEW task_id: 58e011e8-9eec-412b-b483-298487a002d8
36+
INFO: 7
37+
DEBUG: NEW task_id: 6957e51c-d8e4-4787-9adf-d5960a46f2b0
38+
INFO: Waiting for results... of resources: 8
39+
DEBUG: Formatted endpoint: /api2/json/version
40+
DEBUG: Formatted endpoint: /api2/json/version
41+
DEBUG: Formatted endpoint: /api2/json/version
42+
DEBUG: Formatted endpoint: /api2/json/version
43+
DEBUG: Formatted endpoint: /api2/json/version
44+
DEBUG: Formatted endpoint: /api2/json/version
45+
DEBUG: Formatted endpoint: /api2/json/version
46+
DEBUG: Formatted endpoint: /api2/json/version
47+
INFO: ['8.3.2', '8.3.2', '8.3.2', '8.3.2', '8.3.2', '8.3.2', '8.3.2', '8.3.2']
48+
```
49+
50+
#### Example: Multiple Parallel with many API Instances (https in Async Mode)
51+
In this example, multiple API instances with reusing the same backend session are created and used in parallel for asynchronous operations:
52+
```python
53+
async def async_main():
54+
register_backends("https")
55+
ext_api = ProxmoxAPI(backend_name="https", backend_type="async")
56+
async with ext_api as api:
57+
print(await api.version.get(filter_keys="version"))
58+
print(await api.cluster.ha.groups.get())
59+
nodes = await api.nodes.get(filter_keys=["node", "status"])
60+
if nodes:
61+
nodes = sorted([n.get("node") for n in nodes if n.get("status") == "online"])
62+
tasks = []
63+
backend = api.backend
64+
for node in nodes:
65+
new_api = ProxmoxAPI(backend=backend)
66+
tasks.append(new_api.nodes(node).status.get(filter_keys=["kversion", "cpuinfo", "memory.total", "uptime"]))
67+
results = await asyncio.gather(*tasks)
68+
for node, data in zip(nodes, results):
69+
print(data)
70+
71+
asyncio.run(async_main())
72+
```
73+
This example demonstrates the ability to handle multiple API instances, enabling efficient parallel operations while reusing the same backend session. This approach is ideal for scenarios where data from multiple nodes needs to be fetched concurrently.
74+
75+
#### Example of Low-Level Requests (Async)
76+
77+
For more advanced use cases, you can perform low-level API requests directly:
78+
79+
##### Solution 1: Prepare and Execute the Request Manually
80+
81+
```python
82+
# Extract the request parameters for the desired endpoint
83+
params = api.version.get(get_request_param=True)
84+
85+
# Perform the asynchronous request using the extracted parameters
86+
response = await api.async_request(**params)
87+
88+
# Analyze the response to filter and extract specific data
89+
print(api._response_analyze(response, filter_keys="version"))
90+
```
91+
92+
##### Solution 2: Simplified Execution with Built-in API Method
93+
```python
94+
# Use the internal execution method directly with request parameters
95+
print(await api._async_execute(params=params, filter_keys="version"))
96+
```
97+
98+
#### Example of Low-Level Parallel Requests Using the Same API Instance (Async)
99+
100+
Perform parallel requests for multiple nodes while reusing the same API instance:
101+
102+
```python
103+
# Prepare tasks for parallel execution
104+
tasks = []
105+
for node in nodes:
106+
print(node) # Log or display the current node
107+
params = api.nodes(node).status.get(get_request_param=True)
108+
109+
# Add the task to the list using the internal execution method
110+
tasks.append(
111+
api._async_execute(
112+
params=params,
113+
filter_keys=["kversion", "cpuinfo", "memory.total", "uptime"],
114+
)
115+
)
116+
117+
# Wait for all tasks to complete and collect results
118+
print("Waiting for results... Number of resources:", len(tasks))
119+
results = await asyncio.gather(*tasks)
120+
```
121+
122+
#### Example of Parallel Requests Using the Queue of pre created API Instances in Thread Pool (Sync)
123+
124+
```python
125+
import queue
126+
from concurrent.futures import ThreadPoolExecutor
127+
128+
from ext_api.backends.registry import register_backends
129+
from ext_api.proxmox_api import ProxmoxAPI
130+
MAX_THREADS = 4
131+
132+
register_backends("https")
133+
clients = [ProxmoxAPI(backend_name="https") for _ in range(MAX_THREADS)]
134+
client_queue = queue.Queue()
135+
136+
# Populate the queue with clients
137+
for c in clients:
138+
client_queue.put(c)
139+
140+
def get_version():
141+
client = client_queue.get()
142+
try:
143+
# Get an available client from the queue
144+
with client as api:
145+
response = api.version.get()
146+
return response
147+
finally:
148+
# Return the client to the queue
149+
client_queue.put(client)
150+
151+
tasks = []
152+
with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
153+
for _ in range(8):
154+
print(f"Task submit: {len(tasks)}")
155+
tasks.append(executor.submit(get_version))
156+
print("futures created")
157+
results = [task.result() for task in tasks]
158+
print(results)
159+
```
160+
Results:
161+
```log
162+
DEBUG: Creating backend: https of type: sync
163+
DEBUG: Creating backend: https of type: sync
164+
DEBUG: Creating backend: https of type: sync
165+
DEBUG: Creating backend: https of type: sync
166+
DEBUG: Task submit: 0
167+
DEBUG: Task submit: 1
168+
DEBUG: Task submit: 2
169+
DEBUG: Task submit: 3
170+
DEBUG: Task submit: 4
171+
DEBUG: Task submit: 5
172+
DEBUG: Task submit: 6
173+
DEBUG: Task submit: 7
174+
DEBUG: NEW task_id: 6f8c5d7b-52ac-42db-bf1e-3e3314ac36a9
175+
DEBUG: Formatted endpoint: /api2/json/version
176+
DEBUG: NEW task_id: eb986b66-77ef-41f5-9282-e627fea07407
177+
DEBUG: NEW task_id: 61950227-42ff-4923-b16c-a565d8430394
178+
DEBUG: NEW task_id: 45359f3c-9821-4ff4-99c1-1338b396cd48
179+
DEBUG: Formatted endpoint: /api2/json/version
180+
DEBUG: Formatted endpoint: /api2/json/version
181+
DEBUG: Formatted endpoint: /api2/json/version
182+
DEBUG: NEW task_id: 838ab521-5687-4ae1-8c73-6bd7f9567b66
183+
DEBUG: Formatted endpoint: /api2/json/version
184+
DEBUG: NEW task_id: 2332882c-1ab9-48da-98bf-c54295c37761
185+
DEBUG: Formatted endpoint: /api2/json/version
186+
DEBUG: NEW task_id: 6d4770b3-0a04-4f4f-8b10-373112b1fae9
187+
DEBUG: Formatted endpoint: /api2/json/version
188+
DEBUG: NEW task_id: d79ebba9-581e-46a0-a076-b607ae4012e4
189+
DEBUG: Formatted endpoint: /api2/json/version
190+
futures created
191+
['8.3.2', '8.3.2', '8.3.2', '8.3.2', '8.3.2', '8.3.2', '8.3.2', '8.3.2']
192+
```
193+
194+
These examples provide flexibility for advanced API usage, allowing you to control request preparation and execution explicitly, even in parallel scenarios.

0 commit comments

Comments
 (0)