Showing content from https://github.com/onflow/atree/releases/tag/v0.8.0 below:
Release v0.8.0 ยท onflow/atree ยท GitHub
Atree v0.8.0 introduces payload inlining and metadata deduplication to improve memory use, storage size, and speed of various servers that use payloads (not limited to EN). This improves network stability and performance as we handle future data growth.
Inlining and deduplication reduces payload count and mtrie nodes. This reduces size and growth rate of execution state.
- Inlining stores small arrays and maps (previously stored in their own payload) into an existing payload.
- Deduplication removes redundant metadata (e.g. Cadence struct field names and type information) to reduce payload size.
This release passed multiple types of tests, including 1034 hours of smoke tests that concluded on August 19, 2024.
Results exceeded goals mentioned in Scaling Execution Node (forum announcement).
Note
Sept. 4, 2024: ๐ Atree inlined 500 million payloads (-61%) and eliminated over 1 billion mtrie nodes on mainnet!
Impact of these payload and mtrie node reductions include:
- Reduced RAM use by hundreds of GB on each mainnet execution node.
- Reduced SSD storage use (e.g. checkpoint file sizes).
- Reduced network bandwidth for chunk data packs (mentioned by Peter).
- Speedup transactions by 5-8% (mentioned by Jan).
- Speedup network upgrades (e.g. Cadence 1.0 migrations, future migrations).
- Future efficiency of other servers, components, databases (files/cache/index), etc. that benefit from fewer payloads.
Sept. 30, 2024: ๐ Beyond execution nodes, bluesign shared a 5x speedup in tinyAN bootstrap time!
"btw amazing work this atree inlining, my tinyAN bootstrap time improved like 5x"
Future: We can replace nonatree domain payloads with atree payloads to reduce mainnet payload count by another 20-30%.
While devnet (shown below) had better percentage improvements, mainnet eliminated more mtrie nodes (over 1 billion)!
Memory Reduction Before/After Atree Inlining & Deduplication (Devnet Aug 14, 2024)
Mtrie Nodes and Atree Payloads (Aug 14, 2024 Devnet) Atree Payload Sizes (bytes) Total Counts Before After Change mtrie nodes 1,182,195,945 343,949,052 -838,246,893 atree payloads 455,399,623 102,283,931 -353,115,692 Total Sizes Before After Change mtrie nodes 113.49 GB 33.02 GB -80.47 GB atree payloads 87.74 GB 69.28 GB -18.46 GB combined 201.23 GB 102.30 GB -98.93 GB Before After count 455,399,623 102,283,931 mean 192.7 677.4 std 996.4 2,065.7 min 13 22 25% 88 199 50% 139 665 75% 206 1,043 99% 876 1,267 99.5% 1,078 1,453 max 5,853,892 5,853,892 What's Changed (since v0.6.0)
- Change
OrderedMap.Get() to return Value instead of Storable by @fxamacker in #318
- Change
Array.Get() to return Value instead of Storable by @fxamacker in #316
- Optimize max map value size to reduce number of registers by @fxamacker in #314
- Refactor to remove lint warnings by @fxamacker in #319
- Add
Array.ID() and OrderedMap.ID() by @fxamacker in #321
- Rename ID to ValueID by @fxamacker in #325
- Refactor creating new StorableSlab by @fxamacker in #324
- Unexport
SlabID fields to prevent misuse by @fxamacker in #323
- Rename StorageID related types, vars, and funcs by @fxamacker in #322
- Add support for iterating only loaded values by @fxamacker in #311
- Refactor
hasPointer for register inlining by @fxamacker in #327
- Reduce size of encoded array slab and bump version by @fxamacker in #330
- Add ignore filters to dependabot.yml by @fxamacker in #302
- Reduce encoded size of map and bump version by @fxamacker in #331
- Update README.md to describe Atree by @fxamacker in #332
- Update responsible disclosure link by @jribbink in #333
- Fix slab size when resetting mutable storable in OrderedMap by @fxamacker in #337
- Omit empty next slab ID in encoded map data slab by @fxamacker in #340
- Omit empty next slab ID in encoded array data slab by @fxamacker in #339
- Fix slab size when resetting mutable storable in Array by @fxamacker in #336
- Refactor encoding version and flag to add more flags by @fxamacker in #338
- Add readonly iterators and support value mutations only from non-readonly iterators by @fxamacker in #345
- Atree Register Inlining and Data Deduplication by @fxamacker in #342
- Update smoke test for atree inlining by @fxamacker in #348
- Update for Cadence integration for atree inlining and deduplication by @fxamacker in #352
- Make smoke tests check recently added data deduplication feature by @fxamacker in #350
- Remove ContainerStorable.EncodeAsElement by @fxamacker in #354
- Add feature to support mutation for array and map iterators by @fxamacker in #359
- Add support for changing type info of atree arrays (atree inlining branch) by @fxamacker in #376
- Add support for changing type info of atree maps (atree inlining branch) by @fxamacker in #377
- Reduce RAM and persistent storage by deduplicating inlined dict type info by @fxamacker in #369
- Use encoded type info to deduplicate extra data by @fxamacker in #381
- Fix error type for external errors during serialization by @fxamacker in #382
- Add PersistentSlabStorage.GetAllChildReferences() for atree inlining by @fxamacker in #392
- Add feature to enable atree inlining migration to fix references to non-existent registers by @fxamacker in #388
- Fix migration filter for old unreferenced slabs (atree inlining feature branch) by @fxamacker in #396
- Add NonderterministicFastCommit to speed up migrations when ordering isn't required by @fxamacker in #403
- Add BatchPreload to decode slabs in parallel and cache by @fxamacker in #404
- Check mutation of elements from readonly map iterator by @fxamacker in #410
- Check mutation of elements from readonly array iterator by @fxamacker in #411
- Update comment for NondeterministicFastCommit by @fxamacker in #412
- Update copyright notice to Flow Foundation by @fxamacker in #415
- Update copyright notice to Flow Foundation (in main branch) by @fxamacker in #417
- Update feature/array-map-inlining (atree inlining feature branch) by @fxamacker in #428
- Merge
feature/array-map-inlining (atree inlining feature branch) to main by @fxamacker in #429
- Add functions to check availablility of CBOR tag numbers by @fxamacker in #434
- Add SlabID.Address() by @fxamacker in #435
- Make smoke tests check if CBOR tag nums are reserved by @fxamacker in #436
Changes to CI and version bumps...
Special thanks (alphabetically):
- Bastian for feature requests, approving proposed designs, numerous PR reviews, and feedback! ๐
- Dete for suggestion to deduplicate data when encoding (in 2021)! ๐
- Ramtin for suggestion to inline atree registers, approving the proposed designs, and feedback! ๐ช
New Contributors
Full Changelog: v0.6.0...v0.8.0
RetroSearch is an open source project built by @garambo
| Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4
