IRC meeting summary for 2017-02-23
- Git and SHA1 collisions
- Bitcoin Core 0.14.0 release
- Travis CI issue
- Code reorganization
Git and SHA1 Collisions
A few hours before the beginning of the meeting, several researchers announced that they had produced the first case of two different files both having the same hash (a situation called a hash collision) when using the SHA1 hash function.
The Git revision control system used by Bitcoin Core and many other projects uses SHA1 hashes to allow people to ensure that they all have the exact same code. The ability to generate SHA1 collisions undermines that assurance.
Many members of the security community have been requesting Git change from SHA1 to SHA256 or another more-secure hash function for years, but the Git developers have strongly resisted this change (probably because it is backwards incompatible, meaning that it’s likely that all Git users will need to upgrade at roughly the same time).
Discussion initially focused on clarifying the severity of the situation and then moved towards describing potential ways the Bitcoin Core project can deal with the problem while everyone waits for the Git developers to release a more secure program. Solutions proposed included:
“I wonder how hard it is to create an overlay that goes back in history, computes a sha256 for every tree and commit, and then include gpg signatures on those?” (Pieter Wuille)
“We can update our dev scripts to do a sha256sum of all committed files and sign that as well […] eg the merge scripts could include a hash of sha256sum * in the commit message” (Matt Corallo). Gregory Maxwell proposed something very similar at roughly the same time.
“I have a patch for git to use [sha1collissiondetect] as the hash function and abort() if it detects a potentially-bad hash.” (Matt Corallo)
No explicit conclusion was reached, but presumably developers will investigate some or all of the solutions proposed above while Git developers also work to upgrade their program.
Bitcoin Core 0.14.0 release
Note, this heading covers two related topics from the meeting, (1) “help cfields with adding performance-related release notes” and (2) “RC2 status”.
Bitcoin Core 0.14.0’s first Release Candidate (RC1) was distributed after the previous week’s meeting. No major problems were found but some minor fixes and features remain to be merged.
Many of the major upgrades in 0.14.0 are performance improvements.
Wladimir van der Laan asked those present to suggest ways to quantify the performance improvements so Cory Fields could perform any suggested measurements and add the results to the release notes in PR#9787.
Gregory Maxwell suggested, “sync with last release takes N hours, sync with new release takes Y.”
Fields agreed, “ok, I’ll add a vague 0.13 vs 0.14 sync time. The 0.13 will take time though, I haven’t had the patience to finish one yet.”
Jeremy Rubin suggested using Amazon EC2 as a testing environment but Wladimir van der Laan cautioned, “EC is not a good comparison environment: sloooow I/O”. Fields replied that, “I used EC for my sync benching because I figured it represented a very typical use-case.”
Discussion moved to the second Release Candidate (RC2), which seemed to be ready to tag in Git except for a minor translations update and PR#9829 which was ready to merge but had failing Travis Continuous Integration (Travis CI) tests.
Cory Fields will perform a test of the initial sync on EC2 (this has since been completed, Bitcoin Core 0.14.0 performed 5.7 times faster at intial sync than Bitcoin Core 0.13.2). The tests for PR#9829 were restarted after its author, Russell Yanofsky, indicated that he thought they failed because of an intermittent Travis CI problem.
Subsequent to the meeting, RC2 was tagged.
Travis CI issues
Bitcoin Core development uses the Travis CI testing platform to run the project’s tests against each Pull Request (PR). The tests are only meant to fail when there’s a problem with the code in a PR, but sometimes the tests fail for reasons related to Travis CI. In the past these problems have included bugs on Travis as well as Bitcoin Core reaching Travis’s resource limits.
About a week before the meeting, the testing executable
test_bitcoin began to fail intermittently during Travis CI tests. Issue 9825 was opened to diagnose the problem and work on solutions.
The meeting comments focused on how the testing code could create stack traces for the build log as well as how the tests could be changed to get executables, core dumps, and other build and testing artifacts out of Travis for analysis by developers.
Keep investigating the failures, isolate the problem, and work on fixing it.
Many economic full nodes use Bitcoin Core to enforce the consensus rules. Other programs would benefit from reusing the same code Bitcoin Core uses to ensure they comply with the consensus rules, so several developers have been working over time to make the Bitcoin Core codebase more modular so parts of it can be used in other programs.
Although the goal of making the codebase more modular is well supported by project contributors, actually moving code around tends to break other developer’s pending changes and consumes part of the limited time experts have available to review changes to Bitcoin Core—while providing no new features or performance improvements. This makes code moves a good topic for team discussion so that changes can be agreed upon in advance in a way that minimizes disruption.
Jeremy Rubin began, “I have a [proof of concept] branch which moves most of the pure data structures to a datastructures directory.” He added, “non-bitcoin specific [data structures]. E.g., prevector.”
Gregory Maxwell replied, “I don’t think any of us care to maintain things like prevector for other usage. Making a good library takes a tremendous amount of work.”
Wladimir van der Laan agreed, “I don’t think a bitcoin-datastructures library makes sense. If we offer a library it should be something useful to bitcoin users.”
No explicit conclusion during the meeting (the meeting ran out of time on this topic). Discussion continued after the meeting with Maxwell and Wladimir van der Laan expanding upon their points.
|wumpus||Wladimir van der Laan|
This summary was compiled without input from any of the participants in the discussion, so any errors are the fault of the summary author and not the discussion participants.