Skip to content

Port extension to Zero-Dependency C architecture with 95% coverage an… #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Apaezmx merged 1 commit into from
Apr 28, 2026
Merged

Port extension to Zero-Dependency C architecture with 95% coverage an… #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,7 @@ coverage-report/
scratch/
out/
valgrind.log
build.log
regression.diffs
dist/
*.zip
11 changes: 2 additions & 9 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
MODULE_big = pgproto
UPB_OBJS = $(patsubst %.c,%.o,$(shell find vendor/upb -name "*.c" ! -path "*/conformance/*" ! -path "*/stage0/*" ! -path "*/cmake/*")) \
vendor/upb/reflection/stage0/google/protobuf/descriptor.upb.o

OBJS = src/pgproto.o src/io.o src/registry.o src/navigation.o src/gin.o src/json.o src/mutation.o $(UPB_OBJS) vendor/utf8_range/utf8_range.o
PG_CPPFLAGS = -std=c99 -I. -Ivendor -Ivendor/upb/reflection/stage0 -Ivendor/utf8_range -DUPB_BOOTSTRAP_STAGE=0


OBJS = src/pgproto.o src/io.o src/registry.o src/navigation.o src/gin.o src/json.o src/mutation.o
PG_CPPFLAGS = -std=c99 -I.

EXTENSION = pgproto
DATA = sql/pgproto--1.0.sql
Expand All @@ -14,5 +9,3 @@ REGRESS = pgproto_test
PG_CONFIG ?= pg_config
PGXS := $(shell "$(PG_CONFIG)" --pgxs)
include $(PGXS)


30 changes: 14 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
Native Protocol Buffers (proto3) support for PostgreSQL.

Store your protobuf binary data with the rest of your data. Supports:
- **Zero-Dependency Architecture**: Pure C implementation, no external Protobuf libraries required.
- Schema-aware field extraction without JSONB conversions.
- Custom operators for nested field navigation (`->` and `#>`).
- Substantial storage savings over standard JSONB.
- GIN and standard indexing for fast retrieval.

[![Coverage Status](https://img.shields.io/badge/Coverage-87.0%25-brightgreen.svg)](https://github.com/Apaezmx/pgproto)
[![Coverage Status](https://img.shields.io/badge/Coverage-95.0%25-brightgreen.svg)](https://github.com/Apaezmx/pgproto)


## 📊 Performance Results
Expand All @@ -18,8 +19,8 @@ In benchmarks comparing 100,000 serialized `example.Order` messages against equi
| Metric | Protobuf (`pgproto`) | JSONB (`jsonb`) | Native Relational (Normalized 1:N) | Win |
| :--- | :--- | :--- | :--- | :--- |
| **Storage Size** | **16 MB** | 46 MB | 25 MB | **📊 ~35% smaller than Native, ~65% smaller than JSONB!** |
| **Single-Row Lookup Latency (Indexed)** | 4.2 ms | 12.4 ms | **3.3 ms** | Native is fastest for flat lookups, but `pgproto` is close and much faster than JSONB! |
| **Full Document Retrieval Latency** | **4.2 ms** | 12.4 ms | 32.1 ms | **📈 ~7x faster than Native JOINs for full object fetch!** |
| **Single-Row Lookup Latency (Indexed)** | **3.7 ms** | 12.4 ms | 3.3 ms | Native is fastest for flat lookups, but `pgproto` is close and much faster than JSONB! |
| **Full Document Retrieval Latency** | **3.7 ms** | 12.4 ms | 32.1 ms | **📈 ~7x faster than Native JOINs for full object fetch!** |

### 📈 Large Payload Aggregation Benchmark (1KB)
In separate benchmarks querying 100,000 rows with large 1KB payloads (comparing extraction vs JSONB):
Expand All @@ -28,8 +29,8 @@ In separate benchmarks querying 100,000 rows with large 1KB payloads (comparing

### 📊 Concurrent Load Benchmarks
To simulate production load, we ran queries in parallel to measure average latency:
* **10 Parallel Workers**: `pgproto` average latency was **3.78 ms** vs `jsonb` **6.59 ms** (~42% faster).
* **100 Parallel Workers**: `pgproto` average latency was **5.21 ms** vs `jsonb` **10.49 ms** (~50% faster).
* **10 Parallel Workers**: `pgproto` average latency was **3.72 ms** vs `jsonb` **6.59 ms** (~42% faster).
* **100 Parallel Workers**: `pgproto` average latency was **5.11 ms** vs `jsonb` **10.49 ms** (~50% faster).

> [!NOTE]
> `pgproto` combines the storage efficiency of binary compaction with the query flexibility of JSONB, without the overhead of heavy JOINs or text parsing!
Expand All @@ -45,7 +46,7 @@ To simulate production load, we ran queries in parallel to measure average laten
Compile and install the extension (requires standard `build-essential` and `postgresql-server-dev-*`).

```sh
git clone --recursive https://github.com/protocolbuffers/protobuf/upb pgproto
git clone https://github.com/Apaezmx/pgproto
cd pgproto
make
make install # may need sudo
Expand Down Expand Up @@ -251,35 +252,32 @@ docker-compose -f example/docker-compose.yml exec -u root db make install
# Run tests to generate trace data
docker-compose -f example/docker-compose.yml exec -u postgres db make installcheck

# Capture output (ignores negative hit counter overflows and version mismatches)
docker-compose -f example/docker-compose.yml exec -u postgres db lcov --capture --directory . --output-file coverage.info --ignore-errors negative,inconsistent,version,gcov
# Capture output
docker-compose -f example/docker-compose.yml exec -u postgres db lcov --capture --directory src --output-file coverage.info --ignore-errors negative,inconsistent,version,gcov

# Filter out third_party dependencies to see actual extension coverage
docker-compose -f example/docker-compose.yml exec -u postgres db lcov --remove coverage.info '/workspace/third_party/*' --output-file coverage_filtered.info
# Filter out system headers
docker-compose -f example/docker-compose.yml exec -u postgres db lcov --remove coverage.info '/usr/*' --output-file coverage_filtered.info

# View summary
docker-compose -f example/docker-compose.yml exec -u postgres db lcov --summary coverage_filtered.info
```
> [!NOTE]
> Expected coverage for core extension files (excluding `third_party`) is ~87%.
> If you encounter permission issues when writing `coverage.info` or `regression.out`, you may need to create the files first as root on the host or in the container and give them write permissions for everyone.
> For example: `touch coverage.info && chmod a+rw coverage.info`

> Expected coverage for core extension files is **~95%**.

#### 3. 🧠 Memory Leak Analysis
Run showcase queries through `valgrind` to verify memory safety:
```bash
docker-compose -f example/docker-compose.yml exec -u postgres db valgrind --leak-check=full --log-file=/workspace/valgrind.log psql -U postgres -d pgproto_showcase -f /workspace/example/valgrind_full.sql
```
> [!IMPORTANT]
> Note that this profiles the `psql` client process, not the PostgreSQL server process. To profile the extension itself, you would need to run the PostgreSQL server under Valgrind, which is more complex.
> Note that this profiles the `psql` client process. To profile the extension itself, you would need to run the PostgreSQL server under Valgrind.


---

## 🏗️ Technical Details

For historical design plans, caching mechanisms, and deeper architectural discussion, see [DESIGN.md](file:///usr/local/google/home/paezmartinez/pgproto/DESIGN.md).
For technical design plans, custom binary parsing logic, and architectural discussion, see [src/README.md](file:///usr/local/google/home/paezmartinez/pgproto/src/README.md) and [DESIGN.md](file:///usr/local/google/home/paezmartinez/pgproto/DESIGN.md).


---
Expand Down
Loading
Loading