IRC meeting summary for 2017-05-11
Notes / short topics
- Sipa and Gmaxwell have been working on a proposal to maintain a UTXO commitment hash all the time in a way that does this at a cost of a few microseconds per input and output. This would be useful for making
gettxoutsetinfoinstantaneous, or for syncing from someone else’s UTXO set, or as the basis for a softfork later. There are 3 ways of implementing this, all with different performance and security tradeoffs. A few days after the meeting a mailinglist post was made explaining the different approaches and options.
- Per-txo utxo database
- fee targeting/coin selection overhaul
Per-txo utxo database
Since Bitcoin Core v0.8, we’ve used a per-tx model for the chainstate database and its cache. This means that the database is effectively a map from txids to the list of unspent outputs for that transaction. PR #10195 changes that to a map from outpoints ((txid,index) pairs) to just the individual unspent output for that outpoint.
The original reason for aggregating the outputs per transaction was to save space: this way, we could avoid duplicating the txid and transaction meta across the multiple outputs. However, LevelDB internally uses an encoding that omits repeated prefix bytes in keys, and because of that, duplicating the txids is not very significant.
There are many advantages for using a per-txout model:
- Simpler code.
- Avoiding the CPU overhead of deserializing and serializing the unused outputs.
- More predictable memory usage.
- More easily adaptable to various cache flushing strategies.
- Slightly larger on-disk representation, and sometimes larger in-memory representation (when there are multiple outputs for the same txid in the cache, which becomes optional).
The results of testing are very promising as shown in this graph. Sipa notes the chainstate went from 2.2GB to 2.7GB in the test.
Gmaxwell thinks it should be merged quickly to have more testing in master, but it needs some more testing/review. BlueMatt is halfway through review, cfields made it through review but is not confident enough to ACK it.
There still needs to be a few code cleanups and a better user interface for the upgrade process, as there’s a one-time upgrade from the old database to the new one at startup, which takes a couple of minutes.
- Review PR #10195 (Switch chainstate db and cache to per-txout model)
fee targeting/coin selection overhaul
Coin selection is the algorithm that figures out how to spend the coins you have in your wallet. Which ones should be used to fund the transaction. This is a balance between not creating non-standard transactions (by exceeding certain size, or creating dust outputs), trying to find a low fee configuration, give users as much privacy as possible and shrinking the overall UTXO set size.
There have been many improvements made over the years, but there are still issues that pop up once in a while. Instagibbs is working on changing the fee-targeting algorithm to consider “effective value” of considered inputs instead of simply trying to hit an absolute fee, seeing if it failed, then trying again with the estimated total fee at the end of the loop.
Gmaxwell feels a bit uneasy about the changes while there’s no strategy to sweep dust, he’s worried there’s a potential unintended UTXO count blowup.
As conversation goes more into the general approach of fee targeting and coin selection, Morcos mentions the idea of being “fee-smart”, where it could include more inputs when fees are low.
Wumpus likes the idea of cleaning up UTXOs with negative value during low fee-rates, but adds it would be better if those where not created in the first place. Morcos mentions his PR #9343 which tries to do that.
The design goal for Bitcoin Core is to never create change outputs smaller than 0.01BTC unless the wallet is being almost depleted with the transaction, however that goal is not achieved at all.
A good step forwards would also be to dissect and modularize the coin selection out of wallet.cpp.
- a lot of ideas have come up for approaches to fee targeting and coin selection, to be discussed further outside the meeting.
High priority review
Jonasschnelli would like to add PR #10240 (HD wallet auto-restore functionality) which should be included in 0.15.
|wumpus||Wladimir van der Laan|
This summary was compiled without input from any of the participants in the discussion, so any errors are the fault of the summary author and not the discussion participants.