use a binary format for wallet files

[PiRK]

The Electrum wallet files contain the entire transaction history for a wallet, and thus can reach significant sizes (tens of megabytes). The transaction They are currently saved as JSON text, sometimes encrypted.

This could all be improved significantly by using a more compact data format than JSON for disk storage. A structured binary file would reduce the disk space usage, and using a file format that allows for an easy access to some data without having to read/parse all of it would improve performance significantly.

The following formats come to mind:

• sqlite (database format)
• HDF5 (easy to use data format with built-in indexing, compression....); drawback: dependency on numpy (+300MB packages)

[PiRK]

I did some profiling by tweaking the JSON output format. With a very large wallet (>100 MB), there are some potential gains by writing compact unsorted JSON:

diff --git a/electroncash/storage.py b/electroncash/storage.py
index fd872d1d1..08e685c00 100644
--- a/electroncash/storage.py
+++ b/electroncash/storage.py
@@ -120,8 +120,10 @@ class JsonDB(PrintError):
             return
         s = json.dumps(
             self.data,
-            indent=4 if self.output_pretty_json else None,
-            sort_keys=self.output_pretty_json,
+            indent=None,
+            sort_keys=False,
             cls=util.MyEncoder
         )
         s = self.encrypt_before_writing(s)

sort,indent
| 15.489| |03| [profiler] WalletStorage.write 0.9943

no sort, indent
| 16.008| |03| [profiler] WalletStorage.write 0.8581

no sort, no indent
| 15.378| |05| [profiler] WalletStorage.write 0.4795

[PiRK]

Related discussion: spesmilo#4823