Mastering Bitcoin

Chapter 6. The Bitcoin Network

Bitcoin is structured as a peer-to-peer network architecture on top of the Internet. The term peer-to-peer, or P2P, means that the computers that participate in the network are peers to each other, that they are all equal, that there are no “special” knots, and that all knots share the cargo of providing network services. The network knots interconnect in a mesh network with a “plane” topology. There is no server, no centralized service, and no hierarchy within the network. Knots in a peer-to-peer network both provide and consume services at the same time with reciprocity acting as the incentive for participation. Peer-to-peer networks are inherently resilient, decentralized, and open. The preeminent example of a P2P network architecture was the early Internet itself, where knots on the IP network were equal. Today’s Internet architecture is more hierarchical, but the Internet Protocol still retains its flat-topology essence. Beyond bitcoin, the largest and most successful application of P2P technologies is file sharing with Napster as the pioneer and BitTorrent as the most latest evolution of the architecture.

Bitcoin’s P2P network architecture is much more than a topology choice. Bitcoin is a peer-to-peer digital cash system by design, and the network architecture is both a reflection and a foundation of that core characteristic. Decentralization of control is a core design principle and that can only be achieved and maintained by a vapid, decentralized P2P consensus network.

The term “bitcoin network” refers to the collection of knots running the bitcoin P2P protocol. In addition to the bitcoin P2P protocol, there are other protocols such as Stratum, which are used for mining and lightweight or mobile wallets. These extra protocols are provided by gateway routing servers that access the bitcoin network using the bitcoin P2P protocol, and then extend that network to knots running other protocols. For example, Stratum servers connect Stratum mining knots via the Stratum protocol to the main bitcoin network and bridge the Stratum protocol to the bitcoin P2P protocol. We use the term “extended bitcoin network” to refer to the overall network that includes the bitcoin P2P protocol, pool-mining protocols, the Stratum protocol, and any other related protocols connecting the components of the bitcoin system.

Knots Types and Roles

Albeit knots in the bitcoin P2P network are equal, they may take on different roles depending on the functionality they are supporting. A bitcoin knot is a collection of functions: routing, the blockchain database, mining, and wallet services. A utter knot with all four of these functions is shown in Figure 6-1.

All knots include the routing function to participate in the network and might include other functionality. All knots validate and propagate transactions and blocks, and detect and maintain connections to peers. In the full-node example in Figure 6-1, the routing function is indicated by an orange circle named “Network Routing Knot.”

Some knots, called utter knots, also maintain a accomplish and up-to-date copy of the blockchain. Utter knots can autonomously and authoritatively verify any transaction without outer reference. Some knots maintain only a subset of the blockchain and verify transactions using a method called simplified payment verification , or SPV. These knots are known as SPV or lightweight knots. In the full-node example in the figure, the full-node blockchain database function is indicated by a blue circle named “Utter Blockchain.” In Figure 6-3, SPV knots are drawn without the blue circle, displaying that they do not have a total copy of the blockchain.

Mining knots rival to create fresh blocks by running specialized hardware to solve the proof-of-work algorithm. Some mining knots are also total knots, maintaining a utter copy of the blockchain, while others are lightweight knots participating in pool mining and depending on a pool server to maintain a total knot. The mining function is shown in the utter knot as a black circle named “Miner.”

User wallets might be part of a total knot, as is usually the case with desktop bitcoin clients. Increasingly, many user wallets, especially those running on resource-constrained devices such as smartphones, are SPV knots. The wallet function is shown in Figure 6-1 as a green circle named “Wallet”.

In addition to the main knot types on the bitcoin P2P protocol, there are servers and knots running other protocols, such as specialized mining pool protocols and lightweight client-access protocols.

Figure 6-2 shows the most common knot types on the extended bitcoin network.

The Extended Bitcoin Network

The main bitcoin network, running the bitcoin P2P protocol, consists of inbetween 7,000 and Ten,000 listening knots running various versions of the bitcoin reference client (Bitcoin Core) and a few hundred knots running various other implementations of the bitcoin P2P protocol, such as BitcoinJ, Libbitcoin, and btcd. A petite percentage of the knots on the bitcoin P2P network are also mining knots, contesting in the mining process, validating transactions, and creating fresh blocks. Various large companies interface with the bitcoin network by running full-node clients based on the Bitcoin Core client, with utter copies of the blockchain and a network knot, but without mining or wallet functions. These knots act as network edge routers, permitting various other services (exchanges, wallets, block explorers, merchant payment processing) to be built on top.

The extended bitcoin network includes the network running the bitcoin P2P protocol, described earlier, as well as knots running specialized protocols. Linked to the main bitcoin P2P network are a number of pool servers and protocol gateways that connect knots running other protocols. These other protocol knots are mostly pool mining knots (see Chapter 8) and lightweight wallet clients, which do not carry a utter copy of the blockchain.

Figure 6-3 shows the extended bitcoin network with the various types of knots, gateway servers, edge routers, and wallet clients and the various protocols they use to connect to each other.

Network Discovery

When a fresh knot boots up, it must detect other bitcoin knots on the network in order to participate. To commence this process, a fresh knot must detect at least one existing knot on the network and connect to it. The geographic location of other knots is irrelevant; the bitcoin network topology is not geographically defined. Therefore, any existing bitcoin knots can be selected at random.

To connect to a known peer, knots establish a TCP connection, usually to port eight thousand three hundred thirty three (the port generally known as the one used by bitcoin), or an alternative port if one is provided. Upon establishing a connection, the knot will embark a “handshake” (see Figure 6-4) by transmitting a version message, which contains basic identifying information, including:

(See GitHub for an example of the version network message.)

The peer knot responds with verack to acknowledge and establish a connection, and optionally sends its own version message if it wishes to reciprocate the connection and connect back as a peer.

How does a fresh knot find peers? The very first method is to query DNS using a number of “DNS seeds,” which are DNS servers that provide a list of IP addresses of bitcoin knots. Some of those DNS seeds provide a static list of IP addresses of stable bitcoin listening knots. Some of the DNS seeds are custom-made implementations of Tie (Berkeley Internet Name Daemon) that comeback a random subset from a list of bitcoin knot addresses collected by a crawler or a long-running bitcoin knot. The Bitcoin Core client contains the names of five different DNS seeds. The diversity of ownership and diversity of implementation of the different DNS seeds offers a high level or reliability for the initial bootstrapping process. In the Bitcoin Core client, the option to use the DNS seeds is managed by the option switch -dnsseed (set to one by default, to use the DNS seed).

Alternatively, a bootstrapping knot that knows nothing of the network must be given the IP address of at least one bitcoin knot, after which it can establish connections through further introductions. The command-line argument -seednode can be used to connect to one knot just for introductions, using it as a seed. After the initial seed knot is used to form introductions, the client will disconnect from it and use the freshly discovered peers.

Once one or more connections are established, the fresh knot will send an addr message containing its own IP address to its neighbors. The neighbors will, in turn, forward the addr message to their neighbors, ensuring that the freshly connected knot becomes well known and better connected. Additionally, the freshly connected knot can send getaddr to the neighbors, asking them to come back a list of IP addresses of other peers. That way, a knot can find peers to connect to and advertise its existence on the network for other knots to find it. Figure 6-5 shows the address discovery protocol.

A knot must connect to a few different peers in order to establish diverse paths into the bitcoin network. Paths are not reliable—nodes come and go—and so the knot must proceed to detect fresh knots as it loses old connections as well as assist other knots when they bootstrap. Only one connection is needed to bootstrap, because the very first knot can suggest introductions to its peer knots and those peers can suggest further introductions. It’s also unnecessary and wasteful of network resources to connect to more than a handful of knots. After bootstrapping, a knot will recall its most latest successful peer connections, so that if it is rebooted it can quickly reestablish connections with its former peer network. If none of the former peers react to its connection request, the knot can use the seed knots to bootstrap again.

On a knot running the Bitcoin Core client, you can list the peer connections with the directive getpeerinfo :

To override the automatic management of peers and to specify a list of IP addresses, users can provide the option -connect=<IPAddress> and specify one or more IP addresses. If this option is used, the knot will only connect to the selected IP addresses, instead of discovering and maintaining the peer connections automatically.

If there is no traffic on a connection, knots will periodically send a message to maintain the connection. If a knot has not communicated on a connection for more than ninety minutes, it is assumed to be disconnected and a fresh peer will be sought. Thus, the network dynamically adjusts to transient knots and network problems, and can organically grow and shrink as needed without any central control.

Utter Knots

Total knots are knots that maintain a total blockchain with all transactions. More accurately, they most likely should be called “total blockchain knots.” In the early years of bitcoin, all knots were total knots and presently the Bitcoin Core client is a utter blockchain knot. In the past two years, however, fresh forms of bitcoin clients have been introduced that do not maintain a total blockchain but run as lightweight clients. We’ll examine these in more detail in the next section.

Utter blockchain knots maintain a accomplish and up-to-date copy of the bitcoin blockchain with all the transactions, which they independently build and verify, embarking with the very very first block (genesis block) and building up to the latest known block in the network. A utter blockchain knot can independently and authoritatively verify any transaction without recourse or reliance on any other knot or source of information. The utter blockchain knot relies on the network to receive updates about fresh blocks of transactions, which it then verifies and incorporates into its local copy of the blockchain.

Running a total blockchain knot gives you the unspoiled bitcoin practice: independent verification of all transactions without the need to rely on, or trust, any other systems. It’s effortless to tell if you’re running a total knot because it requires 20+ gigabytes of persistent storage (disk space) to store the total blockchain. If you need a lot of disk and it takes two to three days to sync to the network, you are running a total knot. That is the price of accomplish independence and freedom from central authority.

There are a few alternative implementations of utter blockchain bitcoin clients, built using different programming languages and software architectures. However, the most common implementation is the reference client Bitcoin Core, also known as the Satoshi client. More than 90% of the knots on the bitcoin network run various versions of Bitcoin Core. It is identified as “Satoshi” in the sub-version string sent in the version message and shown by the instruction getpeerinfo as we witnessed earlier; for example, /Satoshi:0.8.6/ .

Exchanging “Inventory”

The very first thing a utter knot will do once it connects to peers is attempt to construct a finish blockchain. If it is a brand-new knot and has no blockchain at all, it only knows one block, the genesis block, which is statically embedded in the client software. Commencing with block #0 (the genesis block), the fresh knot will have to download hundreds of thousands of blocks to synchronize with the network and re-establish the total blockchain.

The process of syncing the blockchain starts with the version message, because that contains BestHeight , a node’s current blockchain height (number of blocks). A knot will see the version messages from its peers, know how many blocks they each have, and be able to compare to how many blocks it has in its own blockchain. Peered knots will exchange a%605.420%%% getblocks message that contains the hash (fingerprint) of the top block on their local blockchain. One of the peers will be able to identify the received hash as belonging to a block that is not at the top, but rather belongs to an older block, thus deducing that its own local blockchain is longer than its peer’s.

The peer that has the longer blockchain has more blocks than the other knot and can identify which blocks the other knot needs in order to “catch up.” It will identify the very first five hundred blocks to share and transmit their hashes using an inv (inventory) message. The knot missing these blocks will then retrieve them, by issuing a series of getdata messages requesting the total block data and identifying the requested blocks using the hashes from the inv message.

Let’s assume, for example, that a knot only has the genesis block. It will then receive an inv message from its peers containing the hashes of the next five hundred blocks in the chain. It will begin requesting blocks from all of its connected peers, spreading the blast and ensuring that it doesn’t overwhelm any peer with requests. The knot keeps track of how many blocks are “in transit” per peer connection, meaning blocks that it has requested but not received, checking that it does not exceed a limit ( MAX_BLOCKS_IN_TRANSIT_PER_PEER ). This way, if it needs a lot of blocks, it will only request fresh ones as previous requests are fulfilled, permitting the peers to control the tempo of updates and not terrific the network. As each block is received, it is added to the blockchain, as we will see in Chapter 7. As the local blockchain is step by step built up, more blocks are requested and received, and the process proceeds until the knot catches up to the rest of the network.

This process of comparing the local blockchain with the peers and retrieving any missing blocks happens any time a knot goes offline for any period of time. Whether a knot has been offline for a few minutes and is missing a few blocks, or a month and is missing a few thousand blocks, it starts by sending getblocks , gets an inv response, and starts downloading the missing blocks. Figure 6-6 shows the inventory and block propagation protocol.

Simplified Payment Verification (SPV) Knots

Not all knots have the capability to store the total blockchain. Many bitcoin clients are designed to run on space- and power-constrained devices, such as smartphones, tablets, or embedded systems. For such devices, a simplified payment verification (SPV) method is used to permit them to operate without storing the utter blockchain. These types of clients are called SPV clients or lightweight clients. As bitcoin adoption surges, the SPV knot is becoming the most common form of bitcoin knot, especially for bitcoin wallets.

SPV knots download only the block headers and do not download the transactions included in each block. The resulting chain of blocks, without transactions, is 1,000 times smaller than the total blockchain. SPV knots cannot construct a utter picture of all the UTXOs that are available for spending because they do not know about all the transactions on the network. SPV knots verify transactions using a slightly different methodology that relies on peers to provide partial views of relevant parts of the blockchain on request.

As an analogy, a utter knot is like a tourist in a strange city, tooled with a detailed map of every street and every address. By comparison, an SPV knot is like a tourist in a strange city asking random strangers for turn-by-turn directions while knowing only one main avenue. Albeit both tourists can verify the existence of a street by visiting it, the tourist without a map doesn’t know what lies down any of the side streets and doesn’t know what other streets exist. Placed in front of twenty three Church Street, the tourist without a map cannot know if there are a dozen other “23 Church Street” addresses in the city and whether this is the right one. The mapless tourist’s best chance is to ask enough people and hope some of them are not attempting to mug him.

Simplified payment verification verifies transactions by reference to their depth in the blockchain instead of their height . Whereas a total blockchain knot will construct a fully verified chain of thousands of blocks and transactions reaching down the blockchain (back in time) all the way to the genesis block, an SPV knot will verify the chain of all blocks (but not all transactions) and link that chain to the transaction of interest.

For example, when examining a transaction in block 300,000, a utter knot links all 300,000 blocks down to the genesis block and builds a utter database of UTXO, establishing the validity of the transaction by confirming that the UTXO remains unspent. An SPV knot cannot validate whether the UTXO is unspent. Instead, the SPV knot will establish a link inbetween the transaction and the block that contains it, using a merkle path (see “Merkle Trees”). Then, the SPV knot waits until it sees the six blocks 300,001 through 300,006 piled on top of the block containing the transaction and verifies it by establishing its depth under blocks 300,006 to 300,001. The fact that other knots on the network accepted block 300,000 and then did the necessary work to produce six more blocks on top of it is proof, by proxy, that the transaction was not a double-spend.

An SPV knot cannot be persuaded that a transaction exists in a block when the transaction does not in fact exist. The SPV knot establishes the existence of a transaction in a block by requesting a merkle path proof and by validating the proof of work in the chain of blocks. However, a transaction’s existence can be “hidden” from an SPV knot. An SPV knot can certainly prove that a transaction exists but cannot verify that a transaction, such as a double-spend of the same UTXO, doesn’t exist because it doesn’t have a record of all transactions. This vulnerability can be used in a denial-of-service attack or for a double-spending attack against SPV knots. To defend against this, an SPV knot needs to connect randomly to several knots, to increase the probability that it is in contact with at least one fair knot. This need to randomly connect means that SPV knots also are vulnerable to network partitioning attacks or Sybil attacks, where they are connected to fake knots or fake networks and do not have access to fair knots or the real bitcoin network.

For most practical purposes, well-connected SPV knots are secure enough, striking the right balance inbetween resource needs, practicality, and security. For infallible security, however, nothing hammers running a total blockchain knot.

A total blockchain knot verifies a transaction by checking the entire chain of thousands of blocks below it in order to ensure that the UTXO is not spent, whereas an SPV knot checks how deep the block is buried by a handful of blocks above it.

To get the block headers, SPV knots use a getheaders message instead of getblocks . The responding peer will send up to Two,000 block headers using a single headers message. The process is otherwise the same as that used by a total knot to retrieve total blocks. SPV knots also set a filter on the connection to peers, to filter the stream of future blocks and transactions sent by the peers. Any transactions of interest are retrieved using a getdata request. The peer generates a tx message containing the transactions, in response. Figure 6-7 shows the synchronization of block headers.

Because SPV knots need to retrieve specific transactions in order to selectively verify them, they also create a privacy risk. Unlike total blockchain knots, which collect all transactions within each block, the SPV node’s requests for specific data can inadvertently expose the addresses in their wallet. For example, a third party monitoring a network could keep track of all the transactions requested by a wallet on an SPV knot and use those to associate bitcoin addresses with the user of that wallet, ruining the user’s privacy.

Shortly after the introduction of SPV/lightweight knots, the bitcoin developers added a feature called bloom filters to address the privacy risks of SPV knots. Bloom filters permit SPV knots to receive a subset of the transactions without exposing precisely which addresses they are interested in, through a filtering mechanism that uses probabilities rather than immobile patterns.

Bloom Filters

A bloom filter is a probabilistic search filter, a way to describe a desired pattern without specifying it exactly. Bloom filters suggest an efficient way to express a search pattern while protecting privacy. They are used by SPV knots to ask their peers for transactions matching a specific pattern, without exposing exactly which addresses they are searching for.

In our previous analogy, a tourist without a map is asking for directions to a specific address, “23 Church St.” If she asks strangers for directions to this street, she inadvertently exposes her destination. A bloom filter is like asking, “Are there any streets in this neighborhood whose name completes in R-C-H?” A question like that exposes slightly less about the desired destination than asking for “23 Church St.” Using this mechanism, a tourist could specify the desired address in more detail as “ending in U-R-C-H” or less detail as “ending in H.” By varying the precision of the search, the tourist exposes more or less information, at the expense of getting more or less specific results. If she asks a less specific pattern, she gets a lot more possible addresses and better privacy, but many of the results are irrelevant. If she asks for a very specific pattern, she gets fewer results but loses privacy.

Bloom filters serve this function by permitting an SPV knot to specify a search pattern for transactions that can be tuned toward precision or privacy. A more specific bloom filter will produce accurate results, but at the expense of exposing what addresses are used in the user’s wallet. A less specific bloom filter will produce more data about more transactions, many irrelevant to the knot, but will permit the knot to maintain better privacy.

An SPV knot will initialize a bloom filter as “empty” and in that state the bloom filter will not match any patterns. The SPV knot will then make a list of all the addresses in its wallet and create a search pattern matching the transaction output that corresponds to each address. Usually, the search pattern is a pay-to-public-key-hash script that is the expected locking script that will be present in any transaction paying to the public-key-hash (address). If the SPV knot is tracking the balance of a P2SH address, the search pattern will be a pay-to-script-hash script, instead. The SPV knot then adds each of the search patterns to the bloom filter, so that the bloom filter can recognize the search pattern if it is present in a transaction. Ultimately, the bloom filter is sent to the peer and the peer uses it to match transactions for transmission to the SPV knot.

Bloom filters are implemented as a variable-size array of N binary digits (a bit field) and a variable number of M hash functions. The hash functions are designed to always produce an output that is inbetween one and N, corresponding to the array of binary digits. The hash functions are generated deterministically, so that any knot implementing a bloom filter will always use the same hash functions and get the same results for a specific input. By choosing different length (N) bloom filters and a different number (M) of hash functions, the bloom filter can be tuned, varying the level of accuracy and therefore privacy.

In Figure 6-8, we use a very puny array of sixteen bits and a set of three hash functions to demonstrate how bloom filters work.

The bloom filter is initialized so that the array of bits is all zeros. To add a pattern to the bloom filter, the pattern is hashed by each hash function in turn. Applying the very first hash function to the input results in a number inbetween one and N. The corresponding bit in the array (indexed from one to N) is found and set to one , thereby recording the output of the hash function. Then, the next hash function is used to set another bit and so on. Once all M hash functions have been applied, the search pattern will be “recorded” in the bloom filter as M bits that have been switched from zero to one .

Figure 6-9 is an example of adding a pattern “A” to the plain bloom filter shown in Figure 6-8.

Adding a 2nd pattern is as ordinary as repeating this process. The pattern is hashed by each hash function in turn and the result is recorded by setting the bits to one . Note that as a bloom filter is packed with more patterns, a hash function result might coincide with a bit that is already set to one , in which case the bit is not switched. In essence, as more patterns record on overlapping bits, the bloom filter starts to become saturated with more bits set to one and the accuracy of the filter decreases. This is why the filter is a probabilistic data structure—it gets less accurate as more patterns are added. The accuracy depends on the number of patterns added versus the size of the bit array (N) and number of hash functions (M). A larger bit array and more hash functions can record more patterns with higher accuracy. A smaller bit array or fewer hash functions will record fewer patterns and produce less accuracy.

Figure 6-10 is an example of adding a 2nd pattern “B” to the ordinary bloom filter.

To test if a pattern is part of a bloom filter, the pattern is hashed by each hash function and the resulting bit pattern is tested against the bit array. If all the bits indexed by the hash functions are set to one , then the pattern is very likely recorded in the bloom filter. Because the bits may be set because of overlap from numerous patterns, the reaction is not certain, but is rather probabilistic. In plain terms, a bloom filter positive match is a “Maybe, Yes.”

Figure 6-11 is an example of testing the existence of pattern “X” in the ordinary bloom filter. The corresponding bits are set to one , so the pattern is very likely a match.

On the contrary, if a pattern is tested against the bloom filter and any one of the bits is set to zero , this proves that the pattern was not recorded in the bloom filter. A negative result is not a probability, it is a certainty. In elementary terms, a negative match on a bloom filter is a “Undoubtedly Not!”

Figure 6-12 is an example of testing the existence of pattern “Y” in the elementary bloom filter. One of the corresponding bits is set to zero , so the pattern is certainly not a match.

Bitcoin’s implementation of bloom filters is described in Bitcoin Improvement Proposal thirty seven (BIP0037). See Appendix B or visit GitHub.

Bloom Filters and Inventory Updates

Bloom filters are used to filter the transactions (and blocks containing them) that an SPV knot receives from its peers. SPV knots will create a filter that matches only the addresses held in the SPV node’s wallet. The SPV knot will then send a filterload message to the peer, containing the bloom filter to use on the connection. After a filter is established, the peer will then test each transaction’s outputs against the bloom filter. Only transactions that match the filter are sent to the knot.

In response to a getdata message from the knot, peers will send a merkleblock message that contains only block headers for blocks matching the filter and a merkle path (see “Merkle Trees”) for each matching transaction. The peer will then also send tx messages containing the transactions matched by the filter.

The knot setting the bloom filter can interactively add patterns to the filter by sending a filteradd message. To clear the bloom filter, the knot can send a filterclear message. Because it is not possible to eliminate a pattern from a bloom filter, a knot has to clear and resend a fresh bloom filter if a pattern is no longer desired.

Transaction Pools

Almost every knot on the bitcoin network maintains a improvised list of unconfirmed transactions called the memory pool , mempool , or transaction pool . Knots use this pool to keep track of transactions that are known to the network but are not yet included in the blockchain. For example, a knot that holds a user’s wallet will use the transaction pool to track incoming payments to the user’s wallet that have been received on the network but are not yet confirmed.

As transactions are received and verified, they are added to the transaction pool and relayed to the neighboring knots to propagate on the network.

Some knot implementations also maintain a separate pool of orphaned transactions. If a transaction’s inputs refer to a transaction that is not yet known, such as a missing parent, the orphan transaction will be stored temporarily in the orphan pool until the parent transaction arrives.

When a transaction is added to the transaction pool, the orphan pool is checked for any orphans that reference this transaction’s outputs (its children). Any matching orphans are then validated. If valid, they are liquidated from the orphan pool and added to the transaction pool, completing the chain that began with the parent transaction. In light of the freshly added transaction, which is no longer an orphan, the process is repeated recursively looking for any further descendants, until no more descendants are found. Through this process, the arrival of a parent transaction triggers a cascade reconstruction of an entire chain of interdependent transactions by re-uniting the orphans with their parents all the way down the chain.

Both the transaction pool and orphan pool (where implemented) are stored in local memory and are not saved on persistent storage; rather, they are dynamically populated from incoming network messages. When a knot starts, both pools are empty and are little by little populated with fresh transactions received on the network.

Some implementations of the bitcoin client also maintain a UTXO database or UTXO pool, which is the set of all unspent outputs on the blockchain. Albeit the name “UTXO pool” sounds similar to the transaction pool, it represents a different set of data. Unlike the transaction and orphan pools, the UTXO pool is not initialized empty but instead contains millions of entries of unspent transaction outputs, including some dating back to 2009. The UTXO pool may be housed in local memory or as an indexed database table on persistent storage.

Whereas the transaction and orphan pools represent a single node’s local perspective and might vary significantly from knot to knot depending upon when the knot was commenced or restarted, the UTXO pool represents the emergent consensus of the network and therefore will vary little inbetween knots. Furthermore, the transaction and orphan pools only contain unconfirmed transactions, while the UTXO pool only contains confirmed outputs.

Alert Messages

Alert messages are a seldom used function, but are nevertheless implemented in most knots. Alert messages are bitcoin’s “emergency broadcast system,” a means by which the core bitcoin developers can send an emergency text message to all bitcoin knots. This feature is implemented to permit the core developer team to notify all bitcoin users of a serious problem in the bitcoin network, such as a critical bug that requires user activity. The alert system has only been used a handful of times, most notably in early two thousand thirteen when a critical database bug caused a multiblock fork to occur in the bitcoin blockchain.

Alert messages are propagated by the alert message. The alert message contains several fields, including:

Alerts are cryptographically signed by a public key. The corresponding private key is held by a few select members of the core development team. The digital signature ensures that fake alerts will not be propagated on the network.

Each knot receiving this alert message will verify it, check for expiration, and propagate it to all its peers, thus ensuring rapid propagation across the entire network. In addition to propagating the alert, the knots might implement a user interface function to present the alert to the user.

In the Bitcoin Core client, the alert is configured with the command-line option -alertnotify , which specifies a directive to run when an alert is received. The alert message is passed as a parameter to the alertnotify guideline. Most commonly, the alertnotify directive is set to generate an email message to the administrator of the knot, containing the alert message. The alert is also displayed as a pop-up dialog in the graphical user interface (bitcoin-Qt) if it is running.

Other implementations of the bitcoin protocol might treat the alert in different ways. Many hardware-embedded bitcoin mining systems do not implement the alert message function because they have no user interface. It is strongly recommended that miners running such mining systems subscribe to alerts via a mining pool operator or by running a lightweight knot just for alert purposes.

Mastering Bitcoin

Chapter 6. The Bitcoin Network

Bitcoin is structured as a peer-to-peer network architecture on top of the Internet. The term peer-to-peer, or P2P, means that the computers that participate in the network are peers to each other, that they are all equal, that there are no “special” knots, and that all knots share the cargo of providing network services. The network knots interconnect in a mesh network with a “plane” topology. There is no server, no centralized service, and no hierarchy within the network. Knots in a peer-to-peer network both provide and consume services at the same time with reciprocity acting as the incentive for participation. Peer-to-peer networks are inherently resilient, decentralized, and open. The preeminent example of a P2P network architecture was the early Internet itself, where knots on the IP network were equal. Today’s Internet architecture is more hierarchical, but the Internet Protocol still retains its flat-topology essence. Beyond bitcoin, the largest and most successful application of P2P technologies is file sharing with Napster as the pioneer and BitTorrent as the most latest evolution of the architecture.

Bitcoin’s P2P network architecture is much more than a topology choice. Bitcoin is a peer-to-peer digital cash system by design, and the network architecture is both a reflection and a foundation of that core characteristic. Decentralization of control is a core design principle and that can only be achieved and maintained by a plane, decentralized P2P consensus network.

The term “bitcoin network” refers to the collection of knots running the bitcoin P2P protocol. In addition to the bitcoin P2P protocol, there are other protocols such as Stratum, which are used for mining and lightweight or mobile wallets. These extra protocols are provided by gateway routing servers that access the bitcoin network using the bitcoin P2P protocol, and then extend that network to knots running other protocols. For example, Stratum servers connect Stratum mining knots via the Stratum protocol to the main bitcoin network and bridge the Stratum protocol to the bitcoin P2P protocol. We use the term “extended bitcoin network” to refer to the overall network that includes the bitcoin P2P protocol, pool-mining protocols, the Stratum protocol, and any other related protocols connecting the components of the bitcoin system.

Knots Types and Roles

Albeit knots in the bitcoin P2P network are equal, they may take on different roles depending on the functionality they are supporting. A bitcoin knot is a collection of functions: routing, the blockchain database, mining, and wallet services. A total knot with all four of these functions is shown in Figure 6-1.

All knots include the routing function to participate in the network and might include other functionality. All knots validate and propagate transactions and blocks, and detect and maintain connections to peers. In the full-node example in Figure 6-1, the routing function is indicated by an orange circle named “Network Routing Knot.”

Some knots, called total knots, also maintain a finish and up-to-date copy of the blockchain. Utter knots can autonomously and authoritatively verify any transaction without outer reference. Some knots maintain only a subset of the blockchain and verify transactions using a method called simplified payment verification , or SPV. These knots are known as SPV or lightweight knots. In the full-node example in the figure, the full-node blockchain database function is indicated by a blue circle named “Utter Blockchain.” In Figure 6-3, SPV knots are drawn without the blue circle, showcasing that they do not have a total copy of the blockchain.

Mining knots challenge to create fresh blocks by running specialized hardware to solve the proof-of-work algorithm. Some mining knots are also total knots, maintaining a total copy of the blockchain, while others are lightweight knots participating in pool mining and depending on a pool server to maintain a total knot. The mining function is shown in the total knot as a black circle named “Miner.”

User wallets might be part of a utter knot, as is usually the case with desktop bitcoin clients. Increasingly, many user wallets, especially those running on resource-constrained devices such as smartphones, are SPV knots. The wallet function is shown in Figure 6-1 as a green circle named “Wallet”.

In addition to the main knot types on the bitcoin P2P protocol, there are servers and knots running other protocols, such as specialized mining pool protocols and lightweight client-access protocols.

Figure 6-2 shows the most common knot types on the extended bitcoin network.

The Extended Bitcoin Network

The main bitcoin network, running the bitcoin P2P protocol, consists of inbetween 7,000 and Ten,000 listening knots running various versions of the bitcoin reference client (Bitcoin Core) and a few hundred knots running various other implementations of the bitcoin P2P protocol, such as BitcoinJ, Libbitcoin, and btcd. A puny percentage of the knots on the bitcoin P2P network are also mining knots, contesting in the mining process, validating transactions, and creating fresh blocks. Various large companies interface with the bitcoin network by running full-node clients based on the Bitcoin Core client, with utter copies of the blockchain and a network knot, but without mining or wallet functions. These knots act as network edge routers, permitting various other services (exchanges, wallets, block explorers, merchant payment processing) to be built on top.

The extended bitcoin network includes the network running the bitcoin P2P protocol, described earlier, as well as knots running specialized protocols. Affixed to the main bitcoin P2P network are a number of pool servers and protocol gateways that connect knots running other protocols. These other protocol knots are mostly pool mining knots (see Chapter 8) and lightweight wallet clients, which do not carry a utter copy of the blockchain.

Figure 6-3 shows the extended bitcoin network with the various types of knots, gateway servers, edge routers, and wallet clients and the various protocols they use to connect to each other.

Network Discovery

When a fresh knot boots up, it must detect other bitcoin knots on the network in order to participate. To embark this process, a fresh knot must detect at least one existing knot on the network and connect to it. The geographic location of other knots is irrelevant; the bitcoin network topology is not geographically defined. Therefore, any existing bitcoin knots can be selected at random.

To connect to a known peer, knots establish a TCP connection, usually to port eight thousand three hundred thirty three (the port generally known as the one used by bitcoin), or an alternative port if one is provided. Upon establishing a connection, the knot will embark a “handshake” (see Figure 6-4) by transmitting a version message, which contains basic identifying information, including:

(See GitHub for an example of the version network message.)

The peer knot responds with verack to acknowledge and establish a connection, and optionally sends its own version message if it wishes to reciprocate the connection and connect back as a peer.

How does a fresh knot find peers? The very first method is to query DNS using a number of “DNS seeds,” which are DNS servers that provide a list of IP addresses of bitcoin knots. Some of those DNS seeds provide a static list of IP addresses of stable bitcoin listening knots. Some of the DNS seeds are custom-made implementations of Tie (Berkeley Internet Name Daemon) that comeback a random subset from a list of bitcoin knot addresses collected by a crawler or a long-running bitcoin knot. The Bitcoin Core client contains the names of five different DNS seeds. The diversity of ownership and diversity of implementation of the different DNS seeds offers a high level or reliability for the initial bootstrapping process. In the Bitcoin Core client, the option to use the DNS seeds is managed by the option switch -dnsseed (set to one by default, to use the DNS seed).

Alternatively, a bootstrapping knot that knows nothing of the network must be given the IP address of at least one bitcoin knot, after which it can establish connections through further introductions. The command-line argument -seednode can be used to connect to one knot just for introductions, using it as a seed. After the initial seed knot is used to form introductions, the client will disconnect from it and use the freshly discovered peers.

Once one or more connections are established, the fresh knot will send an addr message containing its own IP address to its neighbors. The neighbors will, in turn, forward the addr message to their neighbors, ensuring that the freshly connected knot becomes well known and better connected. Additionally, the freshly connected knot can send getaddr to the neighbors, asking them to comeback a list of IP addresses of other peers. That way, a knot can find peers to connect to and advertise its existence on the network for other knots to find it. Figure 6-5 shows the address discovery protocol.

A knot must connect to a few different peers in order to establish diverse paths into the bitcoin network. Paths are not reliable—nodes come and go—and so the knot must proceed to detect fresh knots as it loses old connections as well as assist other knots when they bootstrap. Only one connection is needed to bootstrap, because the very first knot can suggest introductions to its peer knots and those peers can suggest further introductions. It’s also unnecessary and wasteful of network resources to connect to more than a handful of knots. After bootstrapping, a knot will reminisce its most latest successful peer connections, so that if it is rebooted it can quickly reestablish connections with its former peer network. If none of the former peers react to its connection request, the knot can use the seed knots to bootstrap again.

On a knot running the Bitcoin Core client, you can list the peer connections with the guideline getpeerinfo :

To override the automatic management of peers and to specify a list of IP addresses, users can provide the option -connect=<IPAddress> and specify one or more IP addresses. If this option is used, the knot will only connect to the selected IP addresses, instead of discovering and maintaining the peer connections automatically.

If there is no traffic on a connection, knots will periodically send a message to maintain the connection. If a knot has not communicated on a connection for more than ninety minutes, it is assumed to be disconnected and a fresh peer will be sought. Thus, the network dynamically adjusts to transient knots and network problems, and can organically grow and shrink as needed without any central control.

Total Knots

Utter knots are knots that maintain a total blockchain with all transactions. More accurately, they most likely should be called “utter blockchain knots.” In the early years of bitcoin, all knots were utter knots and presently the Bitcoin Core client is a total blockchain knot. In the past two years, however, fresh forms of bitcoin clients have been introduced that do not maintain a utter blockchain but run as lightweight clients. We’ll examine these in more detail in the next section.

Total blockchain knots maintain a accomplish and up-to-date copy of the bitcoin blockchain with all the transactions, which they independently build and verify, kicking off with the very very first block (genesis block) and building up to the latest known block in the network. A utter blockchain knot can independently and authoritatively verify any transaction without recourse or reliance on any other knot or source of information. The total blockchain knot relies on the network to receive updates about fresh blocks of transactions, which it then verifies and incorporates into its local copy of the blockchain.

Running a utter blockchain knot gives you the unspoiled bitcoin practice: independent verification of all transactions without the need to rely on, or trust, any other systems. It’s effortless to tell if you’re running a total knot because it requires 20+ gigabytes of persistent storage (disk space) to store the utter blockchain. If you need a lot of disk and it takes two to three days to sync to the network, you are running a total knot. That is the price of accomplish independence and freedom from central authority.

There are a few alternative implementations of total blockchain bitcoin clients, built using different programming languages and software architectures. However, the most common implementation is the reference client Bitcoin Core, also known as the Satoshi client. More than 90% of the knots on the bitcoin network run various versions of Bitcoin Core. It is identified as “Satoshi” in the sub-version string sent in the version message and shown by the guideline getpeerinfo as we witnessed earlier; for example, /Satoshi:0.8.6/ .

Exchanging “Inventory”

The very first thing a utter knot will do once it connects to peers is attempt to construct a accomplish blockchain. If it is a brand-new knot and has no blockchain at all, it only knows one block, the genesis block, which is statically embedded in the client software. Kicking off with block #0 (the genesis block), the fresh knot will have to download hundreds of thousands of blocks to synchronize with the network and re-establish the utter blockchain.

The process of syncing the blockchain starts with the version message, because that contains BestHeight , a node’s current blockchain height (number of blocks). A knot will see the version messages from its peers, know how many blocks they each have, and be able to compare to how many blocks it has in its own blockchain. Peered knots will exchange a%605.420%%% getblocks message that contains the hash (fingerprint) of the top block on their local blockchain. One of the peers will be able to identify the received hash as belonging to a block that is not at the top, but rather belongs to an older block, thus deducing that its own local blockchain is longer than its peer’s.

The peer that has the longer blockchain has more blocks than the other knot and can identify which blocks the other knot needs in order to “catch up.” It will identify the very first five hundred blocks to share and transmit their hashes using an inv (inventory) message. The knot missing these blocks will then retrieve them, by issuing a series of getdata messages requesting the utter block data and identifying the requested blocks using the hashes from the inv message.

Let’s assume, for example, that a knot only has the genesis block. It will then receive an inv message from its peers containing the hashes of the next five hundred blocks in the chain. It will embark requesting blocks from all of its connected peers, spreading the fountain and ensuring that it doesn’t overwhelm any peer with requests. The knot keeps track of how many blocks are “in transit” per peer connection, meaning blocks that it has requested but not received, checking that it does not exceed a limit ( MAX_BLOCKS_IN_TRANSIT_PER_PEER ). This way, if it needs a lot of blocks, it will only request fresh ones as previous requests are fulfilled, permitting the peers to control the tempo of updates and not tremendous the network. As each block is received, it is added to the blockchain, as we will see in Chapter 7. As the local blockchain is little by little built up, more blocks are requested and received, and the process resumes until the knot catches up to the rest of the network.

This process of comparing the local blockchain with the peers and retrieving any missing blocks happens any time a knot goes offline for any period of time. Whether a knot has been offline for a few minutes and is missing a few blocks, or a month and is missing a few thousand blocks, it starts by sending getblocks , gets an inv response, and starts downloading the missing blocks. Figure 6-6 shows the inventory and block propagation protocol.

Simplified Payment Verification (SPV) Knots

Not all knots have the capability to store the total blockchain. Many bitcoin clients are designed to run on space- and power-constrained devices, such as smartphones, tablets, or embedded systems. For such devices, a simplified payment verification (SPV) method is used to permit them to operate without storing the utter blockchain. These types of clients are called SPV clients or lightweight clients. As bitcoin adoption surges, the SPV knot is becoming the most common form of bitcoin knot, especially for bitcoin wallets.

SPV knots download only the block headers and do not download the transactions included in each block. The resulting chain of blocks, without transactions, is 1,000 times smaller than the utter blockchain. SPV knots cannot construct a utter picture of all the UTXOs that are available for spending because they do not know about all the transactions on the network. SPV knots verify transactions using a slightly different methodology that relies on peers to provide partial views of relevant parts of the blockchain on request.

As an analogy, a total knot is like a tourist in a strange city, tooled with a detailed map of every street and every address. By comparison, an SPV knot is like a tourist in a strange city asking random strangers for turn-by-turn directions while knowing only one main avenue. Albeit both tourists can verify the existence of a street by visiting it, the tourist without a map doesn’t know what lies down any of the side streets and doesn’t know what other streets exist. Placed in front of twenty three Church Street, the tourist without a map cannot know if there are a dozen other “23 Church Street” addresses in the city and whether this is the right one. The mapless tourist’s best chance is to ask enough people and hope some of them are not attempting to mug him.

Simplified payment verification verifies transactions by reference to their depth in the blockchain instead of their height . Whereas a utter blockchain knot will construct a fully verified chain of thousands of blocks and transactions reaching down the blockchain (back in time) all the way to the genesis block, an SPV knot will verify the chain of all blocks (but not all transactions) and link that chain to the transaction of interest.

For example, when examining a transaction in block 300,000, a utter knot links all 300,000 blocks down to the genesis block and builds a total database of UTXO, establishing the validity of the transaction by confirming that the UTXO remains unspent. An SPV knot cannot validate whether the UTXO is unspent. Instead, the SPV knot will establish a link inbetween the transaction and the block that contains it, using a merkle path (see “Merkle Trees”). Then, the SPV knot waits until it sees the six blocks 300,001 through 300,006 piled on top of the block containing the transaction and verifies it by establishing its depth under blocks 300,006 to 300,001. The fact that other knots on the network accepted block 300,000 and then did the necessary work to produce six more blocks on top of it is proof, by proxy, that the transaction was not a double-spend.

An SPV knot cannot be persuaded that a transaction exists in a block when the transaction does not in fact exist. The SPV knot establishes the existence of a transaction in a block by requesting a merkle path proof and by validating the proof of work in the chain of blocks. However, a transaction’s existence can be “hidden” from an SPV knot. An SPV knot can certainly prove that a transaction exists but cannot verify that a transaction, such as a double-spend of the same UTXO, doesn’t exist because it doesn’t have a record of all transactions. This vulnerability can be used in a denial-of-service attack or for a double-spending attack against SPV knots. To defend against this, an SPV knot needs to connect randomly to several knots, to increase the probability that it is in contact with at least one fair knot. This need to randomly connect means that SPV knots also are vulnerable to network partitioning attacks or Sybil attacks, where they are connected to fake knots or fake networks and do not have access to fair knots or the real bitcoin network.

For most practical purposes, well-connected SPV knots are secure enough, striking the right balance inbetween resource needs, practicality, and security. For infallible security, however, nothing hits running a utter blockchain knot.

A total blockchain knot verifies a transaction by checking the entire chain of thousands of blocks below it in order to ensure that the UTXO is not spent, whereas an SPV knot checks how deep the block is buried by a handful of blocks above it.

To get the block headers, SPV knots use a getheaders message instead of getblocks . The responding peer will send up to Two,000 block headers using a single headers message. The process is otherwise the same as that used by a utter knot to retrieve utter blocks. SPV knots also set a filter on the connection to peers, to filter the stream of future blocks and transactions sent by the peers. Any transactions of interest are retrieved using a getdata request. The peer generates a tx message containing the transactions, in response. Figure 6-7 shows the synchronization of block headers.

Because SPV knots need to retrieve specific transactions in order to selectively verify them, they also create a privacy risk. Unlike utter blockchain knots, which collect all transactions within each block, the SPV node’s requests for specific data can inadvertently expose the addresses in their wallet. For example, a third party monitoring a network could keep track of all the transactions requested by a wallet on an SPV knot and use those to associate bitcoin addresses with the user of that wallet, ruining the user’s privacy.

Shortly after the introduction of SPV/lightweight knots, the bitcoin developers added a feature called bloom filters to address the privacy risks of SPV knots. Bloom filters permit SPV knots to receive a subset of the transactions without exposing precisely which addresses they are interested in, through a filtering mechanism that uses probabilities rather than motionless patterns.

Bloom Filters

A bloom filter is a probabilistic search filter, a way to describe a desired pattern without specifying it exactly. Bloom filters suggest an efficient way to express a search pattern while protecting privacy. They are used by SPV knots to ask their peers for transactions matching a specific pattern, without exposing exactly which addresses they are searching for.

In our previous analogy, a tourist without a map is asking for directions to a specific address, “23 Church St.” If she asks strangers for directions to this street, she inadvertently exposes her destination. A bloom filter is like asking, “Are there any streets in this neighborhood whose name completes in R-C-H?” A question like that exposes slightly less about the desired destination than asking for “23 Church St.” Using this technology, a tourist could specify the desired address in more detail as “ending in U-R-C-H” or less detail as “ending in H.” By varying the precision of the search, the tourist exposes more or less information, at the expense of getting more or less specific results. If she asks a less specific pattern, she gets a lot more possible addresses and better privacy, but many of the results are irrelevant. If she asks for a very specific pattern, she gets fewer results but loses privacy.

Bloom filters serve this function by permitting an SPV knot to specify a search pattern for transactions that can be tuned toward precision or privacy. A more specific bloom filter will produce accurate results, but at the expense of exposing what addresses are used in the user’s wallet. A less specific bloom filter will produce more data about more transactions, many irrelevant to the knot, but will permit the knot to maintain better privacy.

An SPV knot will initialize a bloom filter as “empty” and in that state the bloom filter will not match any patterns. The SPV knot will then make a list of all the addresses in its wallet and create a search pattern matching the transaction output that corresponds to each address. Usually, the search pattern is a pay-to-public-key-hash script that is the expected locking script that will be present in any transaction paying to the public-key-hash (address). If the SPV knot is tracking the balance of a P2SH address, the search pattern will be a pay-to-script-hash script, instead. The SPV knot then adds each of the search patterns to the bloom filter, so that the bloom filter can recognize the search pattern if it is present in a transaction. Ultimately, the bloom filter is sent to the peer and the peer uses it to match transactions for transmission to the SPV knot.

Bloom filters are implemented as a variable-size array of N binary digits (a bit field) and a variable number of M hash functions. The hash functions are designed to always produce an output that is inbetween one and N, corresponding to the array of binary digits. The hash functions are generated deterministically, so that any knot implementing a bloom filter will always use the same hash functions and get the same results for a specific input. By choosing different length (N) bloom filters and a different number (M) of hash functions, the bloom filter can be tuned, varying the level of accuracy and therefore privacy.

In Figure 6-8, we use a very petite array of sixteen bits and a set of three hash functions to demonstrate how bloom filters work.

The bloom filter is initialized so that the array of bits is all zeros. To add a pattern to the bloom filter, the pattern is hashed by each hash function in turn. Applying the very first hash function to the input results in a number inbetween one and N. The corresponding bit in the array (indexed from one to N) is found and set to one , thereby recording the output of the hash function. Then, the next hash function is used to set another bit and so on. Once all M hash functions have been applied, the search pattern will be “recorded” in the bloom filter as M bits that have been switched from zero to one .

Figure 6-9 is an example of adding a pattern “A” to the plain bloom filter shown in Figure 6-8.

Adding a 2nd pattern is as elementary as repeating this process. The pattern is hashed by each hash function in turn and the result is recorded by setting the bits to one . Note that as a bloom filter is packed with more patterns, a hash function result might coincide with a bit that is already set to one , in which case the bit is not switched. In essence, as more patterns record on overlapping bits, the bloom filter starts to become saturated with more bits set to one and the accuracy of the filter decreases. This is why the filter is a probabilistic data structure—it gets less accurate as more patterns are added. The accuracy depends on the number of patterns added versus the size of the bit array (N) and number of hash functions (M). A larger bit array and more hash functions can record more patterns with higher accuracy. A smaller bit array or fewer hash functions will record fewer patterns and produce less accuracy.

Figure 6-10 is an example of adding a 2nd pattern “B” to the ordinary bloom filter.

To test if a pattern is part of a bloom filter, the pattern is hashed by each hash function and the resulting bit pattern is tested against the bit array. If all the bits indexed by the hash functions are set to one , then the pattern is most likely recorded in the bloom filter. Because the bits may be set because of overlap from numerous patterns, the response is not certain, but is rather probabilistic. In elementary terms, a bloom filter positive match is a “Maybe, Yes.”

Figure 6-11 is an example of testing the existence of pattern “X” in the ordinary bloom filter. The corresponding bits are set to one , so the pattern is most likely a match.

On the contrary, if a pattern is tested against the bloom filter and any one of the bits is set to zero , this proves that the pattern was not recorded in the bloom filter. A negative result is not a probability, it is a certainty. In plain terms, a negative match on a bloom filter is a “Certainly Not!”

Figure 6-12 is an example of testing the existence of pattern “Y” in the elementary bloom filter. One of the corresponding bits is set to zero , so the pattern is undoubtedly not a match.

Bitcoin’s implementation of bloom filters is described in Bitcoin Improvement Proposal thirty seven (BIP0037). See Appendix B or visit GitHub.

Bloom Filters and Inventory Updates

Bloom filters are used to filter the transactions (and blocks containing them) that an SPV knot receives from its peers. SPV knots will create a filter that matches only the addresses held in the SPV node’s wallet. The SPV knot will then send a filterload message to the peer, containing the bloom filter to use on the connection. After a filter is established, the peer will then test each transaction’s outputs against the bloom filter. Only transactions that match the filter are sent to the knot.

In response to a getdata message from the knot, peers will send a merkleblock message that contains only block headers for blocks matching the filter and a merkle path (see “Merkle Trees”) for each matching transaction. The peer will then also send tx messages containing the transactions matched by the filter.

The knot setting the bloom filter can interactively add patterns to the filter by sending a filteradd message. To clear the bloom filter, the knot can send a filterclear message. Because it is not possible to liquidate a pattern from a bloom filter, a knot has to clear and resend a fresh bloom filter if a pattern is no longer desired.

Transaction Pools

Almost every knot on the bitcoin network maintains a improvised list of unconfirmed transactions called the memory pool , mempool , or transaction pool . Knots use this pool to keep track of transactions that are known to the network but are not yet included in the blockchain. For example, a knot that holds a user’s wallet will use the transaction pool to track incoming payments to the user’s wallet that have been received on the network but are not yet confirmed.

As transactions are received and verified, they are added to the transaction pool and relayed to the neighboring knots to propagate on the network.

Some knot implementations also maintain a separate pool of orphaned transactions. If a transaction’s inputs refer to a transaction that is not yet known, such as a missing parent, the orphan transaction will be stored temporarily in the orphan pool until the parent transaction arrives.

When a transaction is added to the transaction pool, the orphan pool is checked for any orphans that reference this transaction’s outputs (its children). Any matching orphans are then validated. If valid, they are eliminated from the orphan pool and added to the transaction pool, completing the chain that commenced with the parent transaction. In light of the freshly added transaction, which is no longer an orphan, the process is repeated recursively looking for any further descendants, until no more descendants are found. Through this process, the arrival of a parent transaction triggers a cascade reconstruction of an entire chain of interdependent transactions by re-uniting the orphans with their parents all the way down the chain.

Both the transaction pool and orphan pool (where implemented) are stored in local memory and are not saved on persistent storage; rather, they are dynamically populated from incoming network messages. When a knot starts, both pools are empty and are little by little populated with fresh transactions received on the network.

Some implementations of the bitcoin client also maintain a UTXO database or UTXO pool, which is the set of all unspent outputs on the blockchain. Albeit the name “UTXO pool” sounds similar to the transaction pool, it represents a different set of data. Unlike the transaction and orphan pools, the UTXO pool is not initialized empty but instead contains millions of entries of unspent transaction outputs, including some dating back to 2009. The UTXO pool may be housed in local memory or as an indexed database table on persistent storage.

Whereas the transaction and orphan pools represent a single node’s local perspective and might vary significantly from knot to knot depending upon when the knot was commenced or restarted, the UTXO pool represents the emergent consensus of the network and therefore will vary little inbetween knots. Furthermore, the transaction and orphan pools only contain unconfirmed transactions, while the UTXO pool only contains confirmed outputs.

Alert Messages

Alert messages are a seldom used function, but are nevertheless implemented in most knots. Alert messages are bitcoin’s “emergency broadcast system,” a means by which the core bitcoin developers can send an emergency text message to all bitcoin knots. This feature is implemented to permit the core developer team to notify all bitcoin users of a serious problem in the bitcoin network, such as a critical bug that requires user activity. The alert system has only been used a handful of times, most notably in early two thousand thirteen when a critical database bug caused a multiblock fork to occur in the bitcoin blockchain.

Alert messages are propagated by the alert message. The alert message contains several fields, including:

Alerts are cryptographically signed by a public key. The corresponding private key is held by a few select members of the core development team. The digital signature ensures that fake alerts will not be propagated on the network.

Each knot receiving this alert message will verify it, check for expiration, and propagate it to all its peers, thus ensuring rapid propagation across the entire network. In addition to propagating the alert, the knots might implement a user interface function to present the alert to the user.

In the Bitcoin Core client, the alert is configured with the command-line option -alertnotify , which specifies a directive to run when an alert is received. The alert message is passed as a parameter to the alertnotify guideline. Most commonly, the alertnotify instruction is set to generate an email message to the administrator of the knot, containing the alert message. The alert is also displayed as a pop-up dialog in the graphical user interface (bitcoin-Qt) if it is running.

Other implementations of the bitcoin protocol might treat the alert in different ways. Many hardware-embedded bitcoin mining systems do not implement the alert message function because they have no user interface. It is strongly recommended that miners running such mining systems subscribe to alerts via a mining pool operator or by running a lightweight knot just for alert purposes.

Mastering Bitcoin

Chapter 7. The Blockchain

The blockchain data structure is an ordered, back-linked list of blocks of transactions. The blockchain can be stored as a vapid file, or in a elementary database. The Bitcoin Core client stores the blockchain metadata using Google’s LevelDB database. Blocks are linked “back,” each referring to the previous block in the chain. The blockchain is often visualized as a vertical stack, with blocks layered on top of each other and the very first block serving as the foundation of the stack. The visualization of blocks stacked on top of each other results in the use of terms such as “height” to refer to the distance from the very first block, and “top” or “peak” to refer to the most recently added block.

Each block within the blockchain is identified by a hash, generated using the SHA256 cryptographic hash algorithm on the header of the block. Each block also references a previous block, known as the parent block, through the “previous block hash” field in the block header. In other words, each block contains the hash of its parent inwards its own header. The sequence of hashes linking each block to its parent creates a chain going back all the way to the very first block ever created, known as the genesis block .

Albeit a block has just one parent, it can temporarily have numerous children. Each of the children refers to the same block as its parent and contains the same (parent) hash in the “previous block hash” field. Numerous children arise during a blockchain “fork,” a makeshift situation that occurs when different blocks are discovered almost at the same time by different miners (see “Blockchain Forks”). Eventually, only one child block becomes part of the blockchain and the “fork” is resolved. Even however a block may have more than one child, each block can have only one parent. This is because a block has one single “previous block hash” field referencing its single parent.

The “previous block hash” field is inwards the block header and thereby affects the current block’s hash. The child’s own identity switches if the parent’s identity switches. When the parent is modified in any way, the parent’s hash switches. The parent’s switched hash necessitates a switch in the “previous block hash” pointer of the child. This in turn causes the child’s hash to switch, which requires a switch in the pointer of the grandchild, which in turn switches the grandchild, and so on. This cascade effect ensures that once a block has many generations following it, it cannot be switched without forcing a recalculation of all subsequent blocks. Because such a recalculation would require enormous computation, the existence of a long chain of blocks makes the blockchain’s deep history immutable, which is a key feature of bitcoin’s security.

One way to think about the blockchain is like layers in a geological formation, or glacier core sample. The surface layers might switch with the seasons, or even be deepthroated away before they have time to lodge. But once you go a few inches deep, geological layers become more and more stable. By the time you look a few hundred feet down, you are looking at a snapshot of the past that has remained undisturbed for millions of years. In the blockchain, the most latest few blocks might be revised if there is a chain recalculation due to a fork. The top six blocks are like a few inches of topsoil. But once you go more deeply into the blockchain, beyond six blocks, blocks are less and less likely to switch. After one hundred blocks back there is so much stability that the coinbase transaction—the transaction containing freshly mined bitcoins—can be spent. A few thousand blocks back (a month) and the blockchain is lodged history, for all practical purposes. While the protocol always permits a chain to be undone by a longer chain and while the possibility of any block being reversed always exists, the probability of such an event decreases as time passes until it becomes infinitesimal.

Structure of a Block

A block is a container data structure that aggregates transactions for inclusion in the public ledger, the blockchain. The block is made of a header, containing metadata, followed by a long list of transactions that make up the bulk of its size. The block header is eighty bytes, whereas the average transaction is at least two hundred fifty bytes and the average block contains more than five hundred transactions. A accomplish block, with all transactions, is therefore 1,000 times larger than the block header. Table 7-1 describes the structure of a block.

Mastering Bitcoin

Chapter Five. Transactions

Transactions are the most significant part of the bitcoin system. Everything else in bitcoin is designed to ensure that transactions can be created, propagated on the network, validated, and ultimately added to the global ledger of transactions (the blockchain). Transactions are data structures that encode the transfer of value inbetween participants in the bitcoin system. Each transaction is a public entry in bitcoin’s blockchain, the global double-entry bookkeeping ledger.

In this chapter we will examine all the various forms of transactions, what they contain, how to create them, how they are verified, and how they become part of the permanent record of all transactions.

Transaction Lifecycle

A transaction’s lifecycle starts with the transaction’s creation, also known as origination . The transaction is then signed with one or more signatures indicating the authorization to spend the funds referenced by the transaction. The transaction is then broadcast on the bitcoin network, where each network knot (participant) validates and propagates the transaction until it reaches (almost) every knot in the network. Ultimately, the transaction is verified by a mining knot and included in a block of transactions that is recorded on the blockchain.

Once recorded on the blockchain and confirmed by sufficient subsequent blocks (confirmations), the transaction is a permanent part of the bitcoin ledger and is accepted as valid by all participants. The funds allocated to a fresh holder by the transaction can then be spent in a fresh transaction, extending the chain of ownership and beginning the lifecycle of a transaction again.

Creating Transactions

In some ways it helps to think of a transaction in the same way as a paper check. Like a check, a transaction is an instrument that voices the intent to transfer money and is not visible to the financial system until it is submitted for execution. Like a check, the originator of the transaction does not have to be the one signing the transaction.

Transactions can be created online or offline by anyone, even if the person creating the transaction is not an authorized signer on the account. For example, an accounts payable clerk might process payable checks for signature by the CEO. Similarly, an accounts payable clerk can create bitcoin transactions and then have the CEO apply digital signatures to make them valid. Whereas a check references a specific account as the source of the funds, a bitcoin transaction references a specific previous transaction as its source, rather than an account.

Once a transaction has been created, it is signed by the possessor (or owners) of the source funds. If it is decently formed and signed, the signed transaction is now valid and contains all the information needed to execute the transfer of funds. Eventually, the valid transaction has to reach the bitcoin network so that it can be propagated until it reaches a miner for inclusion in the pubic ledger (the blockchain).

Broadcasting Transactions to the Bitcoin Network

Very first, a transaction needs to be delivered to the bitcoin network so that it can be propagated and included in the blockchain. In essence, a bitcoin transaction is just three hundred to four hundred bytes of data and has to reach any one of ems of thousands of bitcoin knots. The senders do not need to trust the knots they use to broadcast the transaction, as long as they use more than one to ensure that it propagates. The knots don’t need to trust the sender or establish the sender’s “identity.” Because the transaction is signed and contains no confidential information, private keys, or credentials, it can be publicly broadcast using any underlying network transport that is convenient. Unlike credit card transactions, for example, which contain sensitive information and can only be transmitted on encrypted networks, a bitcoin transaction can be sent over any network. As long as the transaction can reach a bitcoin knot that will propagate it into the bitcoin network, it doesn’t matter how it is transported to the very first knot.

Bitcoin transactions can therefore be transmitted to the bitcoin network over insecure networks such as WiFi, Bluetooth, NFC, Chirp, barcodes, or by copying and pasting into a web form. In extreme cases, a bitcoin transaction could be transmitted over packet radio, satellite relay, or shortwave using burst transmission, spread spectrum, or frequency hopping to evade detection and jamming. A bitcoin transaction could even be encoded as smileys (emoticons) and posted in a public forum or sent as a text message or Skype talk message. Bitcoin has turned money into a data structure, making it virtually unlikely to stop anyone from creating and executing a bitcoin transaction.

Propagating Transactions on the Bitcoin Network

Once a bitcoin transaction is sent to any knot connected to the bitcoin network, the transaction will be validated by that knot. If valid, that knot will propagate it to the other knots to which it is connected, and a success message will be returned synchronously to the originator. If the transaction is invalid, the knot will reject it and synchronously comeback a rejection message to the originator.

The bitcoin network is a peer-to-peer network, meaning that each bitcoin knot is connected to a few other bitcoin knots that it detects during startup through the peer-to-peer protocol. The entire network forms a loosely connected mesh without a immobilized topology or any structure, making all knots equal peers. Messages, including transactions and blocks, are propagated from each knot to all the peers to which it is connected, a process called “flooding.” A fresh validated transaction injected into any knot on the network will be sent to all of the knots connected to it (neighbors), each of which will send the transaction to all its neighbors, and so on. In this way, within a few seconds a valid transaction will propagate in an exponentially expanding ripple across the network until all knots in the network have received it.

The bitcoin network is designed to propagate transactions and blocks to all knots in an efficient and resilient manner that is resistant to attacks. To prevent spamming, denial-of-service attacks, or other nuisance attacks against the bitcoin system, every knot independently validates every transaction before propagating it further. A malformed transaction will not get beyond one knot. The rules by which transactions are validated are explained in more detail in “Independent Verification of Transactions”.

Transaction Structure

A transaction is a data structure that encodes a transfer of value from a source of funds, called an input , to a destination, called an output . Transaction inputs and outputs are not related to accounts or identities. Instead, you should think of them as bitcoin amounts—chunks of bitcoin—being locked with a specific secret that only the holder, or person who knows the secret, can unlock. A transaction contains a number of fields, as shown in Table 5-1.

Specifies which rules this transaction goes after

1–9 bytes (VarInt)

How many inputs are included

One or more transaction inputs

1–9 bytes (VarInt)

How many outputs are included

One or more transaction outputs

A Unix timestamp or block number

Locktime, also known as nLockTime from the variable name used in the reference client, defines the earliest time that a transaction is valid and can be relayed on the network or added to the blockchain. It is set to zero in most transactions to indicate instantaneous propagation and execution. If locktime is nonzero and below five hundred million, it is interpreted as a block height, meaning the transaction is not valid and is not relayed or included in the blockchain prior to the specified block height. If it is above five hundred million, it is interpreted as a Unix Epoch timestamp (seconds since Jan-1-1970) and the transaction is not valid prior to the specified time. Transactions with locktime specifying a future block or time must be held by the originating system and transmitted to the bitcoin network only after they become valid. The use of locktime is equivalent to postdating a paper check.

Transaction Outputs and Inputs

The fundamental building block of a bitcoin transaction is an unspent transaction output , or UTXO. UTXO are indivisible chunks of bitcoin currency locked to a specific proprietor, recorded on the blockchain, and recognized as currency units by the entire network. The bitcoin network tracks all available (unspent) UTXO presently numbering in the millions. Whenever a user receives bitcoin, that amount is recorded within the blockchain as a UTXO. Thus, a user’s bitcoin might be scattered as UTXO amongst hundreds of transactions and hundreds of blocks. In effect, there is no such thing as a stored balance of a bitcoin address or account; there are only scattered UTXO, locked to specific owners. The concept of a user’s bitcoin balance is a derived construct created by the wallet application. The wallet calculates the user’s balance by scanning the blockchain and aggregating all UTXO belonging to that user.

There are no accounts or balances in bitcoin; there are only unspent transaction outputs (UTXO) scattered in the blockchain.

A UTXO can have an arbitrary value denominated as a numerous of satoshis. Just like dollars can be divided down to two decimal places as cents, bitcoins can be divided down to eight decimal places as satoshis. Albeit UTXO can be any arbitrary value, once created it is indivisible just like a coin that cannot be cut in half. If a UTXO is larger than the desired value of a transaction, it must still be consumed in its entirety and switch must be generated in the transaction. In other words, if you have a twenty bitcoin UTXO and want to pay one bitcoin, your transaction must consume the entire twenty bitcoin UTXO and produce two outputs: one paying one bitcoin to your desired recipient and another paying nineteen bitcoin in switch back to your wallet. As a result, most bitcoin transactions will generate switch.

Imagine a shopper buying a $1.50 beverage, reaching into her wallet and attempting to find a combination of coins and bank notes to cover the $1.50 cost. The shopper will choose exact switch if available (a dollar bill and two quarters), or a combination of smaller denominations (six quarters), or if necessary, a larger unit such as a five dollar bank note. If she arms too much money, say $Five, to the shop holder, she will expect $Three.50 switch, which she will come back to her wallet and have available for future transactions.

Similarly, a bitcoin transaction must be created from a user’s UTXO in whatever denominations that user has available. Users cannot cut a UTXO in half any more than they can cut a dollar bill in half and use it as currency. The user’s wallet application will typically select from the user’s available UTXO various units to compose an amount greater than or equal to the desired transaction amount.

As with real life, the bitcoin application can use several strategies to please the purchase amount: combining several smaller units, finding exact switch, or using a single unit larger than the transaction value and making switch. All of this elaborate assembly of spendable UTXO is done by the user’s wallet automatically and is invisible to users. It is only relevant if you are programmatically constructing raw transactions from UTXO.

The UTXO consumed by a transaction are called transaction inputs, and the UTXO created by a transaction are called transaction outputs. This way, chunks of bitcoin value stir forward from possessor to possessor in a chain of transactions consuming and creating UTXO. Transactions consume UTXO by unlocking it with the signature of the current possessor and create UTXO by locking it to the bitcoin address of the fresh possessor.

The exception to the output and input chain is a special type of transaction called the coinbase transaction, which is the very first transaction in each block. This transaction is placed there by the “winning” miner and creates brand-new bitcoin payable to that miner as a prize for mining. This is how bitcoin’s money supply is created during the mining process, as we will see in Chapter 8.

What comes very first? Inputs or outputs, the chicken or the egg? Stringently speaking, outputs come very first because coinbase transactions, which generate fresh bitcoin, have no inputs and create outputs from nothing.

Transaction Outputs

Every bitcoin transaction creates outputs, which are recorded on the bitcoin ledger. Almost all of these outputs, with one exception (see “Data Output (OP_RETURN)”) create spendable chunks of bitcoin called unspent transaction outputs or UTXO, which are then recognized by the entire network and available for the possessor to spend in a future transaction. Sending someone bitcoin is creating an unspent transaction output (UTXO) registered to their address and available for them to spend.

UTXO are tracked by every full-node bitcoin client as a data set called the UTXO set or UTXO pool , held in a database. Fresh transactions consume (spend) one or more of these outputs from the UTXO set.

Transaction outputs consist of two parts:

  • An amount of bitcoin, denominated in satoshis , the smallest bitcoin unit
  • A locking script , also known as an “encumbrance” that “locks” this amount by specifying the conditions that must be met to spend the output

The transaction scripting language, used in the locking script mentioned previously, is discussed in detail in “Transaction Scripts and Script Language”. Table 5-2 shows the structure of a transaction output.

Bitcoin value in satoshis (Ten -8 bitcoin)

1-9 bytes (VarInt)

Locking-Script length in bytes, to go after

A script defining the conditions needed to spend the output

In Example 5-1, we use the blockchain.info API to find the unspent outputs (UTXO) of a specific address.

Running the script, we see a list of transaction IDs, a colon, the index number of the specific unspent transaction output (UTXO), and the value of that UTXO in satoshis. The locking script is not shown in the output in Example 5-2.

Spending conditions (encumbrances)

Transaction outputs associate a specific amount (in satoshis) to a specific encumbrance or locking script that defines the condition that must be met to spend that amount. In most cases, the locking script will lock the output to a specific bitcoin address, thereby transferring ownership of that amount to the fresh holder. When Alice paid Bob’s Cafe for a cup of coffee, her transaction created a 0.015 bitcoin output encumbered or locked to the cafe’s bitcoin address. That 0.015 bitcoin output was recorded on the blockchain and became part of the Unspent Transaction Output set, meaning it demonstrated in Bob’s wallet as part of the available balance. When Bob chooses to spend that amount, his transaction will release the encumbrance, unlocking the output by providing an unlocking script containing a signature from Bob’s private key.

Transaction Inputs

In plain terms, transaction inputs are pointers to UTXO. They point to a specific UTXO by reference to the transaction hash and sequence number where the UTXO is recorded in the blockchain. To spend UTXO, a transaction input also includes unlocking scripts that please the spending conditions set by the UTXO. The unlocking script is usually a signature proving ownership of the bitcoin address that is in the locking script.

When users make a payment, their wallet constructs a transaction by selecting from the available UTXO. For example, to make a 0.015 bitcoin payment, the wallet app may select a 0.01 UTXO and a 0.005 UTXO, using them both to add up to the desired payment amount.

In Example 5-3, we demonstrate the use of a “greedy” algorithm to select from available UTXO in order to make a specific payment amount. In the example, the available UTXO are provided as a constant array, but in reality, the available UTXO would be retrieved with an RPC call to Bitcoin Core, or to a third-party API as shown in Example 5-1.

If we run the select-utxo.py script without a parameter, it will attempt to construct a set of UTXO (and switch) for a payment of 55,000,000 satoshis (0.55 bitcoin). If you provide a target payment amount as a parameter, the script will select UTXO to make that target payment amount. In Example 5-4, we run the script attempting to make a payment of 0.Five bitcoin or 50,000,000 satoshis.

Once the UTXO is selected, the wallet then produces unlocking scripts containing signatures for each of the UTXO, thereby making them spendable by satisfying their locking script conditions. The wallet adds these UTXO references and unlocking scripts as inputs to the transaction. Table 5-3 shows the structure of a transaction input.

Pointer to the transaction containing the UTXO to be spent

The index number of the UTXO to be spent; very first one is 0

1-9 bytes (VarInt)

Unlocking-Script length in bytes, to go after

A script that fulfills the conditions of the UTXO locking script.

Presently disabled Tx-replacement feature, set to 0xFFFFFFFF

The sequence number is used to override a transaction prior to the expiration of the transaction locktime, which is a feature that is presently disabled in bitcoin. Most transactions set this value to the maximum integer value (0xFFFFFFFF) and it is disregarded by the bitcoin network. If the transaction has a nonzero locktime, at least one of its inputs must have a sequence number below 0xFFFFFFFF in order to enable locktime.

Transaction Fees

Most transactions include transaction fees, which compensate the bitcoin miners for securing the network. Mining and the fees and prizes collected by miners are discussed in more detail in Chapter 8. This section examines how transaction fees are included in a typical transaction. Most wallets calculate and include transaction fees automatically. However, if you are constructing transactions programmatically, or using a command-line interface, you must by hand account for and include these fees.

Transaction fees serve as an incentive to include (mine) a transaction into the next block and also as a disincentive against “spam” transactions or any kind of manhandle of the system, by imposing a puny cost on every transaction. Transaction fees are collected by the miner who mines the block that records the transaction on the blockchain.

Transaction fees are calculated based on the size of the transaction in kilobytes, not the value of the transaction in bitcoin. Overall, transaction fees are set based on market coerces within the bitcoin network. Miners prioritize transactions based on many different criteria, including fees, and might even process transactions for free under certain circumstances. Transaction fees affect the processing priority, meaning that a transaction with sufficient fees is likely to be included in the next-most–mined block, whereas a transaction with insufficient or no fees might be delayed, processed on a best-effort basis after a few blocks, or not processed at all. Transaction fees are not mandatory, and transactions without fees might be processed eventually; however, including transaction fees encourages priority processing.

Over time, the way transaction fees are calculated and the effect they have on transaction prioritization has been evolving. At very first, transaction fees were stationary and constant across the network. Step by step, the fee structure has been relaxed so that it may be influenced by market coerces, based on network capacity and transaction volume. The current minimum transaction fee is immobile at 0.0001 bitcoin or a tenth of a milli-bitcoin per kilobyte, recently decreased from one milli-bitcoin. Most transactions are less than one kilobyte; however, those with numerous inputs or outputs can be larger. In future revisions of the bitcoin protocol, it is expected that wallet applications will use statistical analysis to calculate the most adequate fee to link to a transaction based on the average fees of latest transactions.

The current algorithm used by miners to prioritize transactions for inclusion in a block based on their fees is examined in detail in Chapter 8.

Adding Fees to Transactions

The data structure of transactions does not have a field for fees. Instead, fees are implied as the difference inbetween the sum of inputs and the sum of outputs. Any excess amount that remains after all outputs have been deducted from all inputs is the fee that is collected by the miners.

Transaction fees are implied, as the excess of inputs minus outputs:

This is a somewhat confusing element of transactions and an significant point to understand, because if you are constructing your own transactions you must ensure you do not inadvertently include a very large fee by underspending the inputs. That means that you must account for all inputs, if necessary by creating switch, or you will end up providing the miners a very big peak!

For example, if you consume a 20-bitcoin UTXO to make a 1-bitcoin payment, you must include a 19-bitcoin switch output back to your wallet. Otherwise, the 19-bitcoin “leftover” will be counted as a transaction fee and will be collected by the miner who mines your transaction in a block. Albeit you will receive priority processing and make a miner very glad, this is most likely not what you intended.

If you leave behind to add a switch output in a by hand constructed transaction, you will be paying the switch as a transaction fee. “Keep the switch!” might not be what you intended.

Let’s see how this works in practice, by looking at Alice’s coffee purchase again. Alice wants to spend 0.015 bitcoin to pay for coffee. To ensure this transaction is processed promptly, she will want to include a transaction fee, say 0.001. That will mean that the total cost of the transaction will be 0.016. Her wallet must therefore source a set of UTXO that adds up to 0.016 bitcoin or more and, if necessary, create switch. Let’s say her wallet has a 0.2-bitcoin UTXO available. It will therefore need to consume this UTXO, create one output to Bob’s Cafe for 0.015, and a 2nd output with 0.184 bitcoin in switch back to her own wallet, leaving 0.001 bitcoin unallocated, as an implicit fee for the transaction.

Now let’s look at a different screenplay. Eugenia, our children’s charity director in the Philippines, has ended a fundraiser to purchase school books for the children. She received several thousand petite donations from people all around the world, totaling fifty bitcoin, so her wallet is utter of very puny payments (UTXO). Now she wants to purchase hundreds of school books from a local publisher, paying in bitcoin.

As Eugenia’s wallet application attempts to construct a single larger payment transaction, it must source from the available UTXO set, which is composed of many smaller amounts. That means that the resulting transaction will source from more than a hundred small-value UTXO as inputs and only one output, paying the book publisher. A transaction with that many inputs will be larger than one kilobyte, perhaps two to three kilobytes in size. As a result, it will require a higher fee than the minimal network fee of 0.0001 bitcoin.

Eugenia’s wallet application will calculate the adequate fee by measuring the size of the transaction and multiplying that by the per-kilobyte fee. Many wallets will overpay fees for larger transactions to ensure the transaction is processed promptly. The higher fee is not because Eugenia is spending more money, but because her transaction is more elaborate and larger in size—the fee is independent of the transaction’s bitcoin value.

Transaction Chaining and Orphan Transactions

As we have seen, transactions form a chain, whereby one transaction spends the outputs of the previous transaction (known as the parent) and creates outputs for a subsequent transaction (known as the child). Sometimes an entire chain of transactions depending on each other—say a parent, child, and grandchild transaction—are created at the same time, to fulfill a complicated transactional workflow that requires valid children to be signed before the parent is signed. For example, this is a mechanism used in CoinJoin transactions where numerous parties join transactions together to protect their privacy.

When a chain of transactions is transmitted across the network, they don’t always arrive in the same order. Sometimes, the child might arrive before the parent. In that case, the knots that see a child very first can see that it references a parent transaction that is not yet known. Rather than reject the child, they put it in a makeshift pool to await the arrival of its parent and propagate it to every other knot. The pool of transactions without parents is known as the orphan transaction pool . Once the parent arrives, any orphans that reference the UTXO created by the parent are released from the pool, revalidated recursively, and then the entire chain of transactions can be included in the transaction pool, ready to be mined in a block. Transaction chains can be arbitrarily long, with any number of generations transmitted at the same time. The mechanism of holding orphans in the orphan pool ensures that otherwise valid transactions will not be rejected just because their parent has been delayed and that eventually the chain they belong to is reconstructed in the correct order, regardless of the order of arrival.

There is a limit to the number of orphan transactions stored in memory, to prevent a denial-of-service attack against bitcoin knots. The limit is defined as MAX_ORPHAN_TRANSACTIONS in the source code of the bitcoin reference client. If the number of orphan transactions in the pool exceeds MAX_ORPHAN_TRANSACTIONS , one or more randomly selected orphan transactions are evicted from the pool, until the pool size is back within thresholds.

Transaction Scripts and Script Language

Bitcoin clients validate transactions by executing a script, written in a Forth-like scripting language. Both the locking script (encumbrance) placed on a UTXO and the unlocking script that usually contains a signature are written in this scripting language. When a transaction is validated, the unlocking script in each input is executed alongside the corresponding locking script to see if it sates the spending condition.

Today, most transactions processed through the bitcoin network have the form “Alice pays Bob” and are based on the same script called a Pay-to-Public-Key-Hash script. However, the use of scripts to lock outputs and unlock inputs means that through use of the programming language, transactions can contain an infinite number of conditions. Bitcoin transactions are not limited to the “Alice pays Bob” form and pattern.

This is only the peak of the iceberg of possibilities that can be voiced with this scripting language. In this section, we will demonstrate the components of the bitcoin transaction scripting language and demonstrate how it can be used to express complicated conditions for spending and how those conditions can be sated by unlocking scripts.

Bitcoin transaction validation is not based on a static pattern, but instead is achieved through the execution of a scripting language. This language permits for a almost infinite multitude of conditions to be voiced. This is how bitcoin gets the power of “programmable money.”

Script Construction (Lock + Unlock)

Bitcoin’s transaction validation engine relies on two types of scripts to validate transactions: a locking script and an unlocking script.

A locking script is an encumbrance placed on an output, and it specifies the conditions that must be met to spend the output in the future. Historically, the locking script was called a scriptPubKey , because it usually contained a public key or bitcoin address. In this book we refer to it as a “locking script” to acknowledge the much broader range of possibilities of this scripting technology. In most bitcoin applications, what we refer to as a locking script will show up in the source code as scriptPubKey .

An unlocking script is a script that “solves,” or pleases, the conditions placed on an output by a locking script and permits the output to be spent. Unlocking scripts are part of every transaction input, and most of the time they contain a digital signature produced by the user’s wallet from his or her private key. Historically, the unlocking script is called scriptSig , because it usually contained a digital signature. In most bitcoin applications, the source code refers to the unlocking script as scriptSig . In this book, we refer to it as an “unlocking script” to acknowledge the much broader range of locking script requirements, because not all unlocking scripts must contain signatures.

Every bitcoin client will validate transactions by executing the locking and unlocking scripts together. For each input in the transaction, the validation software will very first retrieve the UTXO referenced by the input. That UTXO contains a locking script defining the conditions required to spend it. The validation software will then take the unlocking script contained in the input that is attempting to spend this UTXO and execute the two scripts.

In the original bitcoin client, the unlocking and locking scripts were concatenated and executed in sequence. For security reasons, this was switched in 2010, because of a vulnerability that permitted a malformed unlocking script to thrust data onto the stack and corrupt the locking script. In the current implementation, the scripts are executed separately with the stack transferred inbetween the two executions, as described next.

Very first, the unlocking script is executed, using the stack execution engine. If the unlocking script executed without errors (e.g., it has no “dangling” operators left over), the main stack (not the alternate stack) is copied and the locking script is executed. If the result of executing the locking script with the stack data copied from the unlocking script is “TRUE,” the unlocking script has succeeded in resolving the conditions imposed by the locking script and, therefore, the input is a valid authorization to spend the UTXO. If any result other than “TRUE” remains after execution of the combined script, the input is invalid because it has failed to please the spending conditions placed on the UTXO. Note that the UTXO is permanently recorded in the blockchain, and therefore is invariable and is unaffected by failed attempts to spend it by reference in a fresh transaction. Only a valid transaction that correctly sates the conditions of the UTXO results in the UTXO being marked as “spent” and liquidated from the set of available (unspent) UTXO.

Figure 5-1 is an example of the unlocking and locking scripts for the most common type of bitcoin transaction (a payment to a public key hash), showcasing the combined script resulting from the concatenation of the unlocking and locking scripts prior to script validation.

Scripting Language

The bitcoin transaction script language, called Script , is a Forth-like reverse-polish notation stack-based execution language. If that sounds like gibberish, you very likely haven’t studied 1960’s programming languages. Script is a very ordinary language that was designed to be limited in scope and executable on a range of hardware, perhaps as plain as an embedded device, such as a handheld calculator. It requires minimal processing and cannot do many of the fancy things modern programming languages can do. In the case of programmable money, that is a deliberate security feature.

Bitcoin’s scripting language is called a stack-based language because it uses a data structure called a stack . A stack is a very elementary data structure, which can be visualized as a stack of cards. A stack permits two operations: thrust and pop. Shove adds an item on top of the stack. Pop liquidates the top item from the stack.

The scripting language executes the script by processing each item from left to right. Numbers (data constants) are shoved onto the stack. Operators shove or pop one or more parameters from the stack, act on them, and might thrust a result onto the stack. For example, OP_ADD will pop two items from the stack, add them, and shove the resulting sum onto the stack.

Conditional operators evaluate a condition, producing a boolean result of TRUE or FALSE. For example, OP_EQUAL pops two items from the stack and thrusts TRUE (TRUE is represented by the number 1) if they are equal or FALSE (represented by zero) if they are not equal. Bitcoin transaction scripts usually contain a conditional operator, so that they can produce the TRUE result that indicates a valid transaction.

In Figure 5-2, the script two Three OP_ADD five OP_EQUAL demonstrates the arithmetic addition operator OP_ADD , adding two numbers and putting the result on the stack, followed by the conditional operator OP_EQUAL , which checks that the resulting sum is equal to five . For brevity, the OP_ prefix is omitted in the step-by-step example.

The following is a slightly more sophisticated script, which calculates two + seven – three + one . Notice that when the script contains several operators in a row, the stack permits the results of one operator to be acted upon by the next operator:

Attempt validating the preceding script yourself using pencil and paper. When the script execution finishes, you should be left with the value TRUE on the stack.

Albeit most locking scripts refer to a bitcoin address or public key, thereby requiring proof of ownership to spend the funds, the script does not have to be that sophisticated. Any combination of locking and unlocking scripts that results in a TRUE value is valid. The elementary arithmetic we used as an example of the scripting language is also a valid locking script that can be used to lock a transaction output.

Use part of the arithmetic example script as the locking script:

which can be sated by a transaction containing an input with the unlocking script:

The validation software combines the locking and unlocking scripts and the resulting script is:

As we witnessed in the step-by-step example in Figure 5-2, when this script is executed, the result is OP_TRUE , making the transaction valid. Not only is this a valid transaction output locking script, but the resulting UTXO could be spent by anyone with the arithmetic abilities to know that the number two pleases the script.

Transactions are valid if the top result on the stack is TRUE (noted as <0x01>), any other non-zero value or if the stack is empty after script execution. Transactions are invalid if the top value on the stack is FALSE (a zero-length empty value, noted as <> ) or if script execution is halted explicitly by an operator, such as OP_VERIFY, OP_RETURN, or a conditional terminator such as OP_ENDIF. See Appendix A for details.

Turing Incompleteness

The bitcoin transaction script language contains many operators, but is deliberately limited in one significant way—there are no loops or sophisticated flow control capabilities other than conditional flow control. This ensures that the language is not Turing Accomplish , meaning that scripts have limited complexity and predictable execution times. Script is not a general-purpose language. These limitations ensure that the language cannot be used to create an infinite loop or other form of “logic bomb” that could be embedded in a transaction in a way that causes a denial-of-service attack against the bitcoin network. Reminisce, every transaction is validated by every total knot on the bitcoin network. A limited language prevents the transaction validation mechanism from being used as a vulnerability.

Stateless Verification

The bitcoin transaction script language is stateless, in that there is no state prior to execution of the script, or state saved after execution of the script. Therefore, all the information needed to execute a script is contained within the script. A script will predictably execute the same way on any system. If your system verifies a script, you can be sure that every other system in the bitcoin network will also verify the script, meaning that a valid transaction is valid for everyone and everyone knows this. This predictability of outcomes is an essential benefit of the bitcoin system.

Standard Transactions

In the very first few years of bitcoin’s development, the developers introduced some limitations in the types of scripts that could be processed by the reference client. These limitations are encoded in a function called isStandard() , which defines five types of “standard” transactions. These limitations are makeshift and might be lifted by the time you read this. Until then, the five standard types of transaction scripts are the only ones that will be accepted by the reference client and most miners who run the reference client. Albeit it is possible to create a nonstandard transaction containing a script that is not one of the standard types, you must find a miner who does not go after these limitations to mine that transaction into a block.

Check the source code of the Bitcoin Core client (the reference implementation) to see what is presently permitted as a valid transaction script.

The five standard types of transaction scripts are pay-to-public-key-hash (P2PKH), public-key, multi-signature (limited to fifteen keys), pay-to-script-hash (P2SH), and data output (OP_RETURN), which are described in more detail in the following sections.

Pay-to-Public-Key-Hash (P2PKH)

The vast majority of transactions processed on the bitcoin network are P2PKH transactions. These contain a locking script that encumbers the output with a public key hash, more commonly known as a bitcoin address. Transactions that pay a bitcoin address contain P2PKH scripts. An output locked by a P2PKH script can be unlocked (spent) by presenting a public key and a digital signature created by the corresponding private key.

For example, let’s look at Alice’s payment to Bob’s Cafe again. Alice made a payment of 0.015 bitcoin to the cafe’s bitcoin address. That transaction output would have a locking script of the form:

The Cafe Public Key Hash is equivalent to the bitcoin address of the cafe, without the Base58Check encoding. Most applications would display the public key hash in hexadecimal encoding and not the familiar bitcoin address Base58Check format that commences with a “1”.

The preceding locking script can be pleased with an unlocking script of the form:

The two scripts together would form the following combined validation script:

When executed, this combined script will evaluate to TRUE if, and only if, the unlocking script matches the conditions set by the locking script. In other words, the result will be TRUE if the unlocking script has a valid signature from the cafe’s private key that corresponds to the public key hash set as an encumbrance.

Figures 5-3 and 5-4 demonstrate (in two parts) a step-by-step execution of the combined script, which will prove this is a valid transaction.

Pay-to-Public-Key

Pay-to-public-key is a simpler form of a bitcoin payment than pay-to-public-key-hash. With this script form, the public key itself is stored in the locking script, rather than a public-key-hash as with P2PKH earlier, which is much shorter. Pay-to-public-key-hash was invented by Satoshi to make bitcoin addresses shorter, for ease of use. Pay-to-public-key is now most often seen in coinbase transactions, generated by older mining software that has not been updated to use P2PKH.

A pay-to-public-key locking script looks like this:

The corresponding unlocking script that must be introduced to unlock this type of output is a ordinary signature, like this:

The combined script, which is validated by the transaction validation software, is:

This script is a ordinary invocation of the CHECKSIG operator, which validates the signature as belonging to the correct key and comebacks TRUE on the stack.

Multi-Signature

Multi-signature scripts set a condition where N public keys are recorded in the script and at least M of those must provide signatures to release the encumbrance. This is also known as an M-of-N scheme, where N is the total number of keys and M is the threshold of signatures required for validation. For example, a 2-of-3 multi-signature is one where three public keys are listed as potential signers and at least two of those must be used to create signatures for a valid transaction to spend the funds. At this time, standard multi-signature scripts are limited to at most fifteen listed public keys, meaning you can do anything from a 1-of-1 to a 15-of-15 multi-signature or any combination within that range. The limitation to fifteen listed keys might be lifted by the time this book is published, so check the isStandard() function to see what is presently accepted by the network.

The general form of a locking script setting an M-of-N multi-signature condition is:

where N is the total number of listed public keys and M is the threshold of required signatures to spend the output.

A locking script setting a 2-of-3 multi-signature condition looks like this:

The preceding locking script can be sated with an unlocking script containing pairs of signatures and public keys:

or any combination of two signatures from the private keys corresponding to the three listed public keys.

The prefix OP_0 is required because of a bug in the original implementation of CHECKMULTISIG where one item too many is popped off the stack. It is disregarded by CHECKMULTISIG and is simply a placeholder.

The two scripts together would form the combined validation script:

When executed, this combined script will evaluate to TRUE if, and only if, the unlocking script matches the conditions set by the locking script. In this case, the condition is whether the unlocking script has a valid signature from the two private keys that correspond to two of the three public keys set as an encumbrance.

Data Output (OP_RETURN)

Bitcoin’s distributed and timestamped ledger, the blockchain, has potential uses far beyond payments. Many developers have attempted to use the transaction scripting language to take advantage of the security and resilience of the system for applications such as digital notary services, stock certificates, and clever contracts. Early attempts to use bitcoin’s script language for these purposes involved creating transaction outputs that recorded data on the blockchain; for example, to record a digital fingerprint of a file in such a way that anyone could establish proof-of-existence of that file on a specific date by reference to that transaction.

The use of bitcoin’s blockchain to store data unrelated to bitcoin payments is a controversial subject. Many developers consider such use abusive and want to discourage it. Others view it as a demonstration of the powerful capabilities of blockchain technology and want to encourage such experimentation. Those who object to the inclusion of non-payment data argue that it causes “blockchain bloat,” burdening those running total bitcoin knots with carrying the cost of disk storage for data that the blockchain was not intended to carry. Moreover, such transactions create UTXO that cannot be spent, using the destination bitcoin address as a free-form 20-byte field. Because the address is used for data, it doesn’t correspond to a private key and the resulting UTXO can never be spent; it’s a fake payment. These transactions that can never be spent are therefore never liquidated from the UTXO set and cause the size of the UTXO database to forever increase, or “bloat.”

In version 0.9 of the Bitcoin Core client, a compromise was reached with the introduction of the OP_RETURN operator. OP_RETURN permits developers to add forty bytes of nonpayment data to a transaction output. However, unlike the use of “fake” UTXO, the OP_RETURN operator creates an explicitly provably unspendable output, which does not need to be stored in the UTXO set. OP_RETURN outputs are recorded on the blockchain, so they consume disk space and contribute to the increase in the blockchain’s size, but they are not stored in the UTXO set and therefore do not bloat the UTXO memory pool and cargo total knots with the cost of more expensive RAM.

OP_RETURN scripts look like this:

The data portion is limited to forty bytes and most often represents a hash, such as the output from the SHA256 algorithm (32 bytes). Many applications put a prefix in front of the data to help identify the application. For example, the Proof of Existence digital notarization service uses the 8-byte prefix “DOCPROOF,” which is ASCII encoded as 44f4350524f4f46 in hexadecimal.

Keep in mind that there is no “unlocking script” that corresponds to OP_RETURN that could possibly be used to “spend” an OP_RETURN output. The entire point of OP_RETURN is that you can’t spend the money locked in that output, and therefore it does not need to be held in the UTXO set as potentially spendable— OP_RETURN is provably un-spendable . OP_RETURN is usually an output with a zero bitcoin amount, because any bitcoin assigned to such an output is effectively lost forever. If an OP_RETURN is encountered by the script validation software, it results instantly in halting the execution of the validation script and marking the transaction as invalid. Thus, if you accidentally reference an OP_RETURN output as an input in a transaction, that transaction is invalid.

A standard transaction (one that conforms to the isStandard() checks) can have only one OP_RETURN output. However, a single OP_RETURN output can be combined in a transaction with outputs of any other type.

Pay-to-Script-Hash (P2SH)

Pay-to-script-hash (P2SH) was introduced in two thousand twelve as a powerful fresh type of transaction that greatly simplifies the use of sophisticated transaction scripts. To explain the need for P2SH, let’s look at a practical example.

In Chapter one we introduced Mohammed, an electronics importer based in Dubai. Mohammed’s company uses bitcoin’s multi-signature feature extensively for its corporate accounts. Multi-signature scripts are one of the most common uses of bitcoin’s advanced scripting capabilities and are a very powerful feature. Mohammed’s company uses a multi-signature script for all customer payments, known in accounting terms as “accounts receivable,” or AR. With the multi-signature scheme, any payments made by customers are locked in such a way that they require at least two signatures to release, from Mohammed and one of his fucking partners or from his attorney who has a backup key. A multi-signature scheme like that offers corporate governance controls and protects against theft, embezzlement, or loss.

The resulting script is fairly long and looks like this:

Albeit multi-signature scripts are a powerful feature, they are cumbersome to use. Given the preceding script, Mohammed would have to communicate this script to every customer prior to payment. Each customer would have to use special bitcoin wallet software with the capability to create custom-made transaction scripts, and each customer would have to understand how to create a transaction using custom-made scripts. Furthermore, the resulting transaction would be about five times larger than a ordinary payment transaction, because this script contains very long public keys. The cargo of that extra-large transaction would be borne by the customer in the form of fees. Ultimately, a large transaction script like this would be carried in the UTXO set in RAM in every total knot, until it was spent. All of these issues make using elaborate output scripts difficult in practice.

Pay-to-script-hash (P2SH) was developed to resolve these practical difficulties and to make the use of sophisticated scripts as effortless as a payment to a bitcoin address. With P2SH payments, the sophisticated locking script is substituted with its digital fingerprint, a cryptographic hash. When a transaction attempting to spend the UTXO is introduced later, it must contain the script that matches the hash, in addition to the unlocking script. In elementary terms, P2SH means “pay to a script matching this hash, a script that will be introduced later when this output is spent.”

In P2SH transactions, the locking script that is substituted by a hash is referred to as the redeem script because it is introduced to the system at redemption time rather than as a locking script. Table 5-4 shows the script without P2SH and Table 5-5 shows the same script encoded with P2SH.

Mastering Bitcoin

Chapter Two. How Bitcoin Works

The bitcoin system, unlike traditional banking and payment systems, is based on de-centralized trust. Instead of a central trusted authority, in bitcoin, trust is achieved as an emergent property from the interactions of different participants in the bitcoin system. In this chapter, we will examine bitcoin from a high level by tracking a single transaction through the bitcoin system and see as it becomes “trusted” and accepted by the bitcoin mechanism of distributed consensus and is ultimately recorded on the blockchain, the distributed ledger of all transactions.

Each example is based on an actual transaction made on the bitcoin network, simulating the interactions inbetween the users (Joe, Alice, and Bob) by sending funds from one wallet to another. While tracking a transaction through the bitcoin network and blockchain, we will use a blockchain explorer site to visualize each step. A blockchain explorer is a web application that operates as a bitcoin search engine, in that it permits you to search for addresses, transactions, and blocks and see the relationships and flows inbetween them.

Popular blockchain explorers include:

Each of these has a search function that can take an address, transaction hash, or block number and find the equivalent data on the bitcoin network and blockchain. With each example, we will provide a URL that takes you directly to the relevant entry, so you can explore it in detail.

Bitcoin Overview

In the overview diagram shown in Figure 2-1, we see that the bitcoin system consists of users with wallets containing keys, transactions that are propagated across the network, and miners who produce (through competitive computation) the consensus blockchain, which is the authoritative ledger of all transactions. In this chapter, we will trace a single transaction as it travels across the network and examine the interactions inbetween each part of the bitcoin system, at a high level. Subsequent chapters will delve into the technology behind wallets, mining, and merchant systems.

Buying a Cup of Coffee

Alice, introduced in the previous chapter, is a fresh user who has just acquired her very first bitcoin. In “Getting Your Very first Bitcoins”, Alice met with her friend Joe to exchange some cash for bitcoin. The transaction created by Joe funded Alice’s wallet with 0.Ten BTC. Now Alice will make her very first retail transaction, buying a cup of coffee at Bob’s coffee shop in Palo Alto, California. Bob’s coffee shop recently began accepting bitcoin payments, by adding a bitcoin option to his point-of-sale system. The prices at Bob’s Cafe are listed in the local currency (US dollars), but at the register, customers have the option of paying in either dollars or bitcoin. Alice places her order for a cup of coffee and Bob comes in the transaction at the register. The point-of-sale system will convert the total price from US dollars to bitcoins at the prevailing market rate and display the prices in both currencies, as well as showcase a QR code containing a payment request for this transaction (see Figure 2-2):

The payment request QR code encodes the following URL, defined in BIP0021:

Unlike a QR code that simply contains a destination bitcoin address, a payment request is a QR-encoded URL that contains a destination address, a payment amount, and a generic description such as “Bob’s Cafe.” This permits a bitcoin wallet application to prefill the information used to send the payment while displaying a human-readable description to the user. You can scan the QR code with a bitcoin wallet application to see what Alice would see.

Bob says, “That’s one-dollar-fifty, or fifteen millibits.”

Alice uses her smartphone to scan the barcode on display. Her smartphone shows a payment of 0.0150 BTC to Bob’s Cafe and she selects Send to authorize the payment. Within a few seconds (about the same amount of time as a credit card authorization), Bob would see the transaction on the register, completing the transaction.

In the following sections we will examine this transaction in more detail, see how Alice’s wallet constructed it, how it was propagated across the network, how it was verified, and eventually, how Bob can spend that amount in subsequent transactions.

The bitcoin network can transact in fractional values, e.g., from milli-bitcoins (1/1000th of a bitcoin) down to 1/100,000,000th of a bitcoin, which is known as a satoshi. Across this book we’ll use the term “bitcoin” to refer to any quantity of bitcoin currency, from the smallest unit (1 satoshi) to the total number (21,000,000) of all bitcoins that will ever be mined.

Bitcoin Transactions

In elementary terms, a transaction tells the network that the possessor of a number of bitcoins has authorized the transfer of some of those bitcoins to another possessor. The fresh possessor can now spend these bitcoins by creating another transaction that authorizes transfer to another possessor, and so on, in a chain of ownership.

Transactions are like lines in a double-entry bookkeeping ledger. In plain terms, each transaction contains one or more “inputs,” which are debits against a bitcoin account. On the other side of the transaction, there are one or more “outputs,” which are credits added to a bitcoin account. The inputs and outputs (debits and credits) do not necessarily add up to the same amount. Instead, outputs add up to slightly less than inputs and the difference represents an implied “transaction fee,” which is a puny payment collected by the miner who includes the transaction in the ledger. A bitcoin transaction is shown as a bookkeeping ledger entry in Figure 2-3.

The transaction also contains proof of ownership for each amount of bitcoin (inputs) whose value is transferred, in the form of a digital signature from the proprietor, which can be independently validated by anyone. In bitcoin terms, “spending” is signing a transaction that transfers value from a previous transaction over to a fresh holder identified by a bitcoin address.

Transactions budge value from transaction inputs to transaction outputs . An input is where the coin value is coming from, usually a previous transaction’s output. A transaction output assigns a fresh proprietor to the value by associating it with a key. The destination key is called an encumbrance . It imposes a requirement for a signature for the funds to be redeemed in future transactions. Outputs from one transaction can be used as inputs in a fresh transaction, thus creating a chain of ownership as the value is moved from address to address (see Figure 2-4).

Alice’s payment to Bob’s Cafe uses a previous transaction as its input. In the previous chapter Alice received bitcoin from her friend Joe in comeback for cash. That transaction has a number of bitcoins locked (encumbered) against Alice’s key. Her fresh transaction to Bob’s Cafe references the previous transaction as an input and creates fresh outputs to pay for the cup of coffee and receive switch. The transactions form a chain, where the inputs from the latest transaction correspond to outputs from previous transactions. Alice’s key provides the signature that unlocks those previous transaction outputs, thereby proving to the bitcoin network that she wields the funds. She fastens the payment for coffee to Bob’s address, thereby “encumbering” that output with the requirement that Bob produces a signature in order to spend that amount. This represents a transfer of value inbetween Alice and Bob. This chain of transactions, from Joe to Alice to Bob, is illustrated in Figure 2-4.

Common Transaction Forms

The most common form of transaction is a ordinary payment from one address to another, which often includes some “switch” returned to the original proprietor. This type of transaction has one input and two outputs and is shown in Figure 2-5.

Another common form of transaction is one that aggregates several inputs into a single output (see Figure 2-6). This represents the real-world equivalent of exchanging a pile of coins and currency notes for a single larger note. Transactions like these are sometimes generated by wallet applications to clean up lots of smaller amounts that were received as switch for payments.

Eventually, another transaction form that is seen often on the bitcoin ledger is a transaction that distributes one input to numerous outputs indicating numerous recipients (see Figure 2-7). This type of transaction is sometimes used by commercial entities to distribute funds, such as when processing payroll payments to numerous employees.

Constructing a Transaction

Alice’s wallet application contains all the logic for selecting adequate inputs and outputs to build a transaction to Alice’s specification. Alice only needs to specify a destination and an amount and the rest happens in the wallet application without her observing the details. Importantly, a wallet application can construct transactions even if it is entirely offline. Like writing a check at home and later sending it to the bank in an envelope, the transaction does not need to be constructed and signed while connected to the bitcoin network. It only has to be sent to the network eventually for it to be executed.

Getting the Right Inputs

Alice’s wallet application will very first have to find inputs that can pay for the amount she wants to send to Bob. Most wallet applications keep a puny database of “unspent transaction outputs” that are locked (encumbered) with the wallet’s own keys. Therefore, Alice’s wallet would contain a copy of the transaction output from Joe’s transaction, which was created in exchange for cash (see “Getting Your Very first Bitcoins”). A bitcoin wallet application that runs as a full-index client actually contains a copy of every unspent output from every transaction in the blockchain. This permits a wallet to construct transaction inputs as well as quickly verify incoming transactions as having correct inputs. However, because a full-index client takes up a lot of disk space, most user wallets run “lightweight” clients that track only the user’s own unspent outputs.

If the wallet application does not maintain a copy of unspent transaction outputs, it can query the bitcoin network to retrieve this information, using a multitude of APIs available by different providers or by asking a full-index knot using the bitcoin JSON RPC API. Example 2-1 shows a RESTful API request, constructed as an HTTP GET guideline to a specific URL. This URL will comeback all the unspent transaction outputs for an address, providing any application the information it needs to construct transaction inputs for spending. We use the ordinary command-line HTTP client cURL to retrieve the response.

The response in Example 2-2 shows one unspent output (one that has not been redeemed yet) under the ownership of Alice’s address 1Cdid9KFAaatwczBwBttQcwXYCpvK8h7FK . The response includes the reference to the transaction in which this unspent output is contained (the payment from Joe) and its value in satoshis, at ten million, equivalent to 0.Ten bitcoin. With this information, Alice’s wallet application can construct a transaction to transfer that value to fresh proprietor addresses.

As you can see, Alice’s wallet contains enough bitcoins in a single unspent output to pay for the cup of coffee. Had this not been the case, Alice’s wallet application might have to “rummage” through a pile of smaller unspent outputs, like picking coins from a purse until it could find enough to pay for coffee. In both cases, there might be a need to get some switch back, which we will see in the next section, as the wallet application creates the transaction outputs (payments).

Creating the Outputs

A transaction output is created in the form of a script that creates an encumbrance on the value and can only be redeemed by the introduction of a solution to the script. In simpler terms, Alice’s transaction output will contain a script that says something like, “This output is payable to whoever can present a signature from the key corresponding to Bob’s public address.” Because only Bob has the wallet with the keys corresponding to that address, only Bob’s wallet can present such a signature to redeem this output. Alice will therefore “encumber” the output value with a request for a signature from Bob.

This transaction will also include a 2nd output, because Alice’s funds are in the form of a 0.Ten BTC output, too much money for the 0.015 BTC cup of coffee. Alice will need 0.085 BTC in switch. Alice’s switch payment is created by Alice’s wallet in the very same transaction as the payment to Bob. Essentially, Alice’s wallet cracks her funds into two payments: one to Bob, and one back to herself. She can then use the switch output in a subsequent transaction, thus spending it later.

Eventually, for the transaction to be processed by the network in a timely style, Alice’s wallet application will add a petite fee. This is not explicit in the transaction; it is implied by the difference inbetween inputs and outputs. If instead of taking 0.085 in switch, Alice creates only 0.0845 as the 2nd output, there will be 0.0005 BTC (half a millibitcoin) left over. The input’s 0.Ten BTC is not fully spent with the two outputs, because they will add up to less than 0.Ten. The resulting difference is the transaction fee that is collected by the miner as a fee for including the transaction in a block and putting it on the blockchain ledger.

The resulting transaction can be seen using a blockchain explorer web application, as shown in Figure 2-8.

Adding the Transaction to the Ledger

The transaction created by Alice’s wallet application is two hundred fifty eight bytes long and contains everything necessary to confirm ownership of the funds and assign fresh owners. Now, the transaction must be transmitted to the bitcoin network where it will become part of the distributed ledger (the blockchain). In the next section we will see how a transaction becomes part of a fresh block and how the block is “mined.” Eventually, we will see how the fresh block, once added to the blockchain, is increasingly trusted by the network as more blocks are added.

Transmitting the transaction

Because the transaction contains all the information necessary to process, it does not matter how or where it is transmitted to the bitcoin network. The bitcoin network is a peer-to-peer network, with each bitcoin client participating by connecting to several other bitcoin clients. The purpose of the bitcoin network is to propagate transactions and blocks to all participants.

How it propagates

Alice’s wallet application can send the fresh transaction to any of the other bitcoin clients it is connected to over any Internet connection: wired, WiFi, or mobile. Her bitcoin wallet does not have to be connected to Bob’s bitcoin wallet directly and she does not have to use the Internet connection suggested by the cafe, tho’ both those options are possible, too. Any bitcoin network knot (other client) that receives a valid transaction it has not seen before will instantly forward it to other knots to which it is connected. Thus, the transaction rapidly propagates out across the peer-to-peer network, reaching a large percentage of the knots within a few seconds.

Bob’s view

If Bob’s bitcoin wallet application is directly connected to Alice’s wallet application, Bob’s wallet application might be the very first knot to receive the transaction. However, even if Alice’s wallet sends the transaction through other knots, it will reach Bob’s wallet within a few seconds. Bob’s wallet will instantly identify Alice’s transaction as an incoming payment because it contains outputs redeemable by Bob’s keys. Bob’s wallet application can also independently verify that the transaction is well formed, uses previously unspent inputs, and contains sufficient transaction fees to be included in the next block. At this point Bob can assume, with little risk, that the transaction will shortly be included in a block and confirmed.

A common misconception about bitcoin transactions is that they must be “confirmed” by waiting ten minutes for a fresh block, or up to sixty minutes for a utter six confirmations. Albeit confirmations ensure the transaction has been accepted by the entire network, such a delay is unnecessary for small-value items such as a cup of coffee. A merchant may accept a valid small-value transaction with no confirmations, with no more risk than a credit card payment made without an ID or a signature, as merchants routinely accept today.

Bitcoin Mining

The transaction is now propagated on the bitcoin network. It does not become part of the collective ledger (the blockchain ) until it is verified and included in a block by a process called mining . See Chapter eight for a detailed explanation.

The bitcoin system of trust is based on computation. Transactions are bundled into blocks , which require an enormous amount of computation to prove, but only a puny amount of computation to verify as proven. The mining process serves two purposes in bitcoin:

  • Mining creates fresh bitcoins in each block, almost like a central bank printing fresh money. The amount of bitcoin created per block is stationary and diminishes with time.
  • Mining creates trust by ensuring that transactions are only confirmed if enough computational power was faithful to the block that contains them. More blocks mean more computation, which means more trust.

A good way to describe mining is like a giant competitive game of sudoku that resets every time someone finds a solution and whose difficulty automatically adjusts so that it takes approximately ten minutes to find a solution. Imagine a giant sudoku puzzle, several thousand rows and columns in size. If I display you a ended puzzle you can verify it fairly quickly. However, if the puzzle has a few squares packed and the rest are empty, it takes a lot of work to solve! The difficulty of the sudoku can be adjusted by switching its size (more or fewer rows and columns), but it can still be verified fairly lightly even if it is very large. The “puzzle” used in bitcoin is based on a cryptographic hash and exhibits similar characteristics: it is asymmetrically hard to solve but effortless to verify, and its difficulty can be adjusted.

In “Bitcoin Uses, Users, and Their Stories”, we introduced Jing, a computer engineering student in Shanghai. Jing is participating in the bitcoin network as a miner. Every ten minutes or so, Jing joins thousands of other miners in a global race to find a solution to a block of transactions. Finding such a solution, the so-called proof of work, requires quadrillions of hashing operations per 2nd across the entire bitcoin network. The algorithm for proof of work involves repeatedly hashing the header of the block and a random number with the SHA256 cryptographic algorithm until a solution matching a predetermined pattern emerges. The very first miner to find such a solution wins the round of competition and publishes that block into the blockchain.

Jing commenced mining in two thousand ten using a very swift desktop computer to find a suitable proof of work for fresh blocks. As more miners commenced joining the bitcoin network, the difficulty of the problem enlargened rapidly. Soon, Jing and other miners upgraded to more specialized hardware, such as high-end dedicated graphical processing units (GPUs) cards such as those used in gaming desktops or consoles. At the time of this writing, the difficulty is so high that it is profitable only to mine with application-specific integrated circuits (ASIC), essentially hundreds of mining algorithms printed in hardware, running in parallel on a single silicon chip. Jing also joined a “mining pool,” which much like a lottery pool permits several participants to share their efforts and the prizes. Jing now runs two USB-connected ASIC machines to mine for bitcoin twenty four hours a day. He pays his electric current costs by selling the bitcoin he is able to generate from mining, creating some income from the profits. His computer runs a copy of bitcoind, the reference bitcoin client, as a backend to his specialized mining software.

Mining Transactions in Blocks

A transaction transmitted across the network is not verified until it becomes part of the global distributed ledger, the blockchain. Every ten minutes on average, miners generate a fresh block that contains all the transactions since the last block. Fresh transactions are permanently flowing into the network from user wallets and other applications. As these are seen by the bitcoin network knots, they get added to a makeshift pool of unverified transactions maintained by each knot. As miners build a fresh block, they add unverified transactions from this pool to a fresh block and then attempt to solve a very hard problem (a.k.a., proof of work) to prove the validity of that fresh block. The process of mining is explained in detail in “Introduction”.

Transactions are added to the fresh block, prioritized by the highest-fee transactions very first and a few other criteria. Each miner starts the process of mining a fresh block of transactions as soon as he receives the previous block from the network, knowing he has lost that previous round of competition. He instantly creates a fresh block, fills it with transactions and the fingerprint of the previous block, and starts calculating the proof of work for the fresh block. Each miner includes a special transaction in his block, one that pays his own bitcoin address a prize of freshly created bitcoins (presently twenty five BTC per block). If he finds a solution that makes that block valid, he “wins” this prize because his successful block is added to the global blockchain and the prize transaction he included becomes spendable. Jing, who participates in a mining pool, has set up his software to create fresh blocks that assign the prize to a pool address. From there, a share of the prize is distributed to Jing and other miners in proportion to the amount of work they contributed in the last round.

Alice’s transaction was picked up by the network and included in the pool of unverified transactions. Because it had sufficient fees, it was included in a fresh block generated by Jing’s mining pool. Approximately five minutes after the transaction was very first transmitted by Alice’s wallet, Jing’s ASIC miner found a solution for the block and published it as block #277316, containing four hundred nineteen other transactions. Jing’s ASIC miner published the fresh block on the bitcoin network, where other miners validated it and embarked the race to generate the next block.

You can see the block that includes Alice’s transaction.

A few minutes later, a fresh block, #277317, is mined by another miner. Because this fresh block is based on the previous block (#277316) that contained Alice’s transaction, it added even more computation on top of that block, thereby strengthening the trust in those transactions. The block containing Alice’s transaction is counted as one “confirmation” of that transaction. Each block mined on top of the one containing the transaction is an extra confirmation. As the blocks pile on top of each other, it becomes exponentially stiffer to switch sides the transaction, thereby making it more and more trusted by the network.

In the diagram in Figure 2-9 we can see block #277316, which contains Alice’s transaction. Below it are 277,316 blocks (including block #0), linked to each other in a chain of blocks (blockchain) all the way back to block #0, known as the genesis block . Over time, as the “height” in blocks increases, so does the computation difficulty for each block and the chain as a entire. The blocks mined after the one that contains Alice’s transaction act as further assurance, as they pile on more computation in a longer and longer chain. By convention, any block with more than six confirmations is considered irrevocable, because it would require an immense amount of computation to invalidate and recalculate six blocks. We will examine the process of mining and the way it builds trust in more detail in Chapter 8.

Mastering Bitcoin

Chapter Trio. The Bitcoin Client

You can download the reference client Bitcoin Core , also known as the “Satoshi client,” from bitcoin.org. The reference client implements all aspects of the bitcoin system, including wallets, a transaction verification engine with a utter copy of the entire transaction ledger (blockchain), and a utter network knot in the peer-to-peer bitcoin network.

On Bitcoin’s Choose Your Wallet page, select Bitcoin Core to download the reference client. Depending on your operating system, you will download an executable installer. For Windows, this is either a ZIP archive or an .exe executable. For Mac OS it is a .dmg disk picture. Linux versions include a PPA package for Ubuntu or a tar.gz archive. The bitcoin.org page that lists recommended bitcoin clients is shown in Figure 3-1.

Running Bitcoin Core for the Very first Time

If you download an installable package, such as an .exe, .dmg, or PPA, you can install it the same way as any application on your operating system. For Windows, run the .exe and go after the step-by-step instructions. For Mac OS, launch the .dmg and haul the Bitcoin-QT icon into your Applications folder. For Ubuntu, double-click the PPA in your File Explorer and it will open the package manager to install the package. Once you have ended installation you should have a fresh application called Bitcoin-Qt in your application list. Double-click the icon to begin the bitcoin client.

The very first time you run Bitcoin Core it will begin downloading the blockchain, a process that might take several days (see Figure 3-2). Leave it running in the background until it displays “Synchronized” and no longer shows “out of sync” next to the balance.

Bitcoin Core keeps a total copy of the transaction ledger (blockchain), with every transaction that has ever occurred on the bitcoin network since its inception in 2009. This dataset is several gigabytes in size (approximately sixteen GB in late 2013) and is downloaded incrementally over several days. The client will not be able to process transactions or update account balances until the total blockchain dataset is downloaded. During that time, the client will display “out of sync” next to the account balances and showcase “Synchronizing” in the footer. Make sure you have enough disk space, bandwidth, and time to accomplish the initial synchronization.

Compiling Bitcoin Core from the Source Code

For developers, there is also the option to download the total source code as a ZIP archive or by cloning the authoritative source repository from GitHub. On the GitHub bitcoin page, select Download ZIP from the sidebar. Alternatively, use the git directive line to create a local copy of the source code on your system. In the following example, we are cloning the source code from a Unix-like instruction line, in Linux or Mac OS:

The instructions and resulting output might vary from version to version. Go after the documentation that comes with the code even if it differs from the instructions you see here, and don’t be astonished if the output displayed on your screen is slightly different from the examples here.

When the git cloning operation has finished, you will have a accomplish local copy of the source code repository in the directory bitcoin . Switch to this directory by typing cd bitcoin at the prompt:

By default, the local copy will be synchronized with the most latest code, which might be an unstable or beta version of bitcoin. Before compiling the code, select a specific version by checking out a release tag . This will synchronize the local copy with a specific snapshot of the code repository identified by a keyword tag. Tags are used by the developers to mark specific releases of the code by version number. Very first, to find the available tags, we use the git tag instruction:

The list of tags shows all the released versions of bitcoin. By convention, release candidates , which are intended for testing, have the suffix “rc”. Stable releases that can be run on production systems have no suffix. From the preceding list, select the highest version release, which at this writing was v0.9.0rc1. To synchronize the local code with this version, use the git checkout instruction:

The source code includes documentation, which can be found in a number of files. Review the main documentation located in README.md in the bitcoin directory by typing more README.md at the prompt and using the space bar to progress to the next page. In this chapter, we will build the command-line bitcoin client, also known as bitcoind on Linux. Review the instructions for compiling the bitcoind command-line client on your platform by typing more doc/build-unix.md . Alternative instructions for Mac OS X and Windows can be found in the doc directory, as build-osx.md or build-msw.md , respectively.

Cautiously review the build prerequisites, which are in the very first part of the build documentation. These are libraries that must be present on your system before you can begin to compile bitcoin. If these prerequisites are missing, the build process will fail with an error. If this happens because you missed a prerequisite, you can install it and then resume the build process from where you left off. Assuming the prerequisites are installed, you commence the build process by generating a set of build scripts using the autogen.sh script.

The Bitcoin Core build process was switched to use the autogen/configure/make system embarking with version 0.9. Older versions use a ordinary Makefile and work slightly differently from the following example. Go after the instructions for the version you want to compile. The autogen/configure/make introduced in 0.9 is likely to be the build system used for all future versions of the code and is the system demonstrated in the following examples.

The autogen.sh script creates a set of automatic configuration scripts that will interrogate your system to detect the correct settings and ensure you have all the necessary libraries to compile the code. The most significant of these is the configure script that offers a number of different options to customize the build process. Type ./configure –help to see the various options:

The configure script permits you to enable or disable certain features of bitcoind through the use of the –enable-FEATURE and –disable-FEATURE flags, where FEATURE is substituted by the feature name, as listed in the help output. In this chapter, we will build the bitcoind client with all the default features. We won’t be using the configuration flags, but you should review them to understand what optional features are part of the client. Next, run the configure script to automatically detect all the necessary libraries and create a customized build script for your system:

If all goes well, the configure instruction will end by creating the customized build scripts that will permit us to compile bitcoind. If there are any missing libraries or errors, the configure instruction will terminate with an error instead of creating the build scripts. If an error occurs, it is most likely because of a missing or incompatible library. Review the build documentation again and make sure you install the missing prerequisites. Then run configure again and see if that fixes the error. Next, you will compile the source code, a process that can take up to an hour to accomplish. During the compilation process you should see output every few seconds or every few minutes, or an error if something goes wrong. The compilation process can be resumed at any time if interrupted. Type make to embark compiling:

If all goes well, bitcoind is now compiled. The final step is to install the bitcoind executable into the system path using the make instruction:

You can confirm that bitcoin is correctly installed by asking the system for the path of the two executables, as goes after:

The default installation of bitcoind puts it in /usr/local/bin . When you very first run bitcoind, it will remind you to create a configuration file with a strong password for the JSON-RPC interface. Run bitcoind by typing bitcoind into the terminal:

Edit the configuration file in your preferred editor and set the parameters, substituting the password with a strong password as recommended by bitcoind. Do not use the password shown here. Create a file inwards the .bitcoin directory so that it is named .bitcoin/bitcoin.conf and inject a username and password:

While you’re editing this configuration file, you might want to set a few other options, such as txindex (see “Transaction Database Index and txindex Option”). For a total listing of the available options, type bitcoind –help .

Now, run the Bitcoin Core client. The very first time you run it, it will rebuild the bitcoin blockchain by downloading all the blocks. This is a multigigabyte file and will take an average of two days to download in total. You can shorten the blockchain initialization time by downloading a partial copy of the blockchain using a BitTorrent client from SourceForge.

Run bitcoind in the background with the option -daemon :

Using Bitcoin Core’s JSON-RPC API from the Guideline Line

The Bitcoin Core client implements a JSON-RPC interface that can also be accessed using the command-line helper bitcoin-cli . The directive line permits us to experiment interactively with the capabilities that are also available programmatically via the API. To embark, invoke the help instruction to see a list of the available bitcoin RPC directives:

Getting Information on the Bitcoin Core Client Status

Bitcoin’s getinfo RPC directive displays basic information about the status of the bitcoin network knot, the wallet, and the blockchain database. Use bitcoin-cli to run it:

The data is returned in JavaScript Object Notation (JSON), a format that can lightly be “consumed” by all programming languages but is also fairly human-readable. Among this data we see the version numbers for the bitcoin software client (90000), protocol (70002), and wallet (60000). We see the current balance contained in the wallet, which is zero. We see the current block height, demonstrating us how many blocks are known to this client (286216). We also see various statistics about the bitcoin network and the settings related to this client. We will explore these settings in more detail in the rest of this chapter.

It will take some time, perhaps more than a day, for the bitcoind client to “catch up” to the current blockchain height as it downloads blocks from other bitcoin clients. You can check its progress using getinfo to see the number of known blocks.

Wallet Setup and Encryption

Directions: encryptwallet , walletpassphrase

Before you proceed with creating keys and other instructions, you should very first encrypt the wallet with a password. For this example, you will use the encryptwallet directive with the password “foo”. Obviously, substitute “foo” with a strong and complicated password!

You can verify the wallet has been encrypted by running getinfo again. This time you will notice a fresh entry called unlocked_until . This is a counter showcasing how long the wallet decryption password will be stored in memory, keeping the wallet unlocked. At very first this will be set to zero, meaning the wallet is locked:

To unlock the wallet, issue the walletpassphrase guideline, which takes two parameters—the password and a number of seconds until the wallet is locked again automatically (a time counter):

You can confirm the wallet is unlocked and see the timeout by running getinfo again:

Wallet Backup, Plain-text Dump, and Restore

Instructions: backupwallet , importwallet , dumpwallet

Next, we will practice creating a wallet backup file and then restoring the wallet from the backup file. Use the backupwallet directive to back up, providing the filename as the parameter. Here we back up the wallet to the file wallet.backup :

Now, to restore the backup file, use the importwallet directive. If your wallet is locked, you will need to unlock it very first (see walletpassphrase in the preceding section) in order to import the backup file:

The dumpwallet directive can be used to dump the wallet into a text file that is human-readable:

Wallet Addresses and Receiving Transactions

Directions: getnewaddress, getreceivedbyaddress, listtransactions, getaddressesbyaccount, getbalance

The bitcoin reference client maintains a pool of addresses, the size of which is displayed by keypoolsize when you use the instruction getinfo . These addresses are generated automatically and can then be used as public receiving addresses or switch addresses. To get one of these addresses, use the getnewaddress guideline:

Now, we can use this address to send a petite amount of bitcoin to our bitcoind wallet from an outer wallet (assuming you have some bitcoin in an exchange, web wallet, or other bitcoind wallet held elsewhere). For this example, we will send fifty millibits (0.050 bitcoin) to the preceding address.

We can now query the bitcoind client for the amount received by this address, and specify how many confirmations are required before an amount is counted in that balance. For this example, we will specify zero confirmations. A few seconds after sending the bitcoin from another wallet, we will see it reflected in the wallet. We use getreceivedbyaddress with the address and the number of confirmations set to zero (0):

If we omit the zero from the end of this guideline, we will only see the amounts that have at least minconf confirmations, where minconf is the setting for the minimum number of confirmations before a transaction is listed in the balance. The minconf setting is specified in the bitcoind configuration file. Because the transaction sending this bitcoin was only sent in the last few seconds, it has still not confirmed and therefore we will see it list a zero balance:

The transactions received by the entire wallet can also be displayed using the listtransactions instruction:

We can list all addresses in the entire wallet using the getaddressesbyaccount guideline:

Ultimately, the guideline getbalance will showcase the total balance of the wallet, adding up all transactions confirmed with at least minconf confirmations:

If the transaction has not yet confirmed, the balance returned by getbalance will be zero. The configuration option “minconf” determines the minimum number of confirmations that are required before a transaction shows in the balance.

Exploring and Decoding Transactions

Instructions: gettransaction , getrawtransaction , decoderawtransaction

We’ll now explore the incoming transaction that was listed previously using the gettransaction instruction. We can retrieve a transaction by its transaction hash, shown at txid earlier, with the gettransaction directive:

Transaction IDs are not authoritative until a transaction has been confirmed. Absence of a transaction hash in the blockchain does not mean the transaction was not processed. This is known as “transaction malleability,” because transaction hashes can be modified prior to confirmation in a block. After confirmation, the txid is immutable and authoritative.

The transaction form shown with the guideline gettransaction is the simplified form. To retrieve the total transaction code and decode it, we will use two guidelines: getrawtransaction and decoderawtransaction . Very first, getrawtransaction takes the transaction hash (txid) as a parameter and comebacks the utter transaction as a “raw” hex string, exactly as it exists on the bitcoin network:

To decode this hex string, use the decoderawtransaction instruction. Copy and paste the hex as the very first parameter of decoderawtransaction to get the total contents interpreted as a JSON data structure (for formatting reasons the hex string is shortened in the following example):

The transaction decode shows all the components of this transaction, including the transaction inputs and outputs. In this case we see that the transaction that credited our fresh address with fifty millibits used one input and generated two outputs. The input to this transaction was the output from a previously confirmed transaction (shown as the vin txid beginning with d3c7 ). The two outputs correspond to the fifty millibit credit and an output with switch back to the sender.

We can further explore the blockchain by examining the previous transaction referenced by its txid in this transaction using the same directives (e.g., gettransaction ). Leaping from transaction to transaction we can go after a chain of transactions back as the coins are transmitted from possessor address to holder address.

Once the transaction we received has been confirmed by inclusion in a block, the gettransaction directive will come back extra information, showcasing the block hash (identifier) in which the transaction was included:

Here, we see the fresh information in the entries blockhash (the hash of the block in which the transaction was included), and blockindex with value eighteen (indicating that our transaction was the 18th transaction in that block).

By default, Bitcoin Core builds a database containing only the transactions related to the user’s wallet. If you want to be able to access any transaction with directives like gettransaction , you need to configure Bitcoin Core to build a finish transaction index, which can be achieved with the txindex option. Set txindex=1 in the Bitcoin Core configuration file (usually found in your home directory under .bitcoin/bitcoin.conf ). Once you switch this parameter, you need to restart bitcoind and wait for it to rebuild the index.

Exploring Blocks

Directives: getblock , getblockhash

Now that we know which block our transaction was included in, we can query that block. We use the getblock instruction with the block hash as the parameter:

The block contains three hundred sixty seven transactions and as you can see, the 18th transaction listed ( 9ca8f9… ) is the txid of the one crediting fifty millibits to our address. The height entry tells us this is the 286384th block in the blockchain.

We can also retrieve a block by its block height using the getblockhash instruction, which takes the block height as the parameter and comes back the block hash for that block:

Here, we retrieve the block hash of the “genesis block,” the very first block mined by Satoshi Nakamoto, at height zero. Retrieving this block shows:

The getblock , getblockhash , and gettransaction directions can be used to explore the blockchain database, programmatically.

Creating, Signing, and Submitting Transactions Based on Unspent Outputs

Guidelines: listunspent , gettxout , createrawtransaction , decoderawtransaction , signrawtransaction , sendrawtransaction

Bitcoin’s transactions are based on the concept of spending “outputs,” which are the result of previous transactions, to create a transaction chain that transfers ownership from address to address. Our wallet has now received a transaction that assigned one such output to our address. Once this is confirmed, we can spend that output.

Very first, we use the listunspent guideline to showcase all the unspent confirmed outputs in our wallet:

We see that the transaction 9ca8f9… created an output (with vout index 0) assigned to the address 1hvzSo… for the amount of fifty millibits, which at this point has received seven confirmations. Transactions use previously created outputs as their inputs by referring to them by the previous txid and vout index. We will now create a transaction that will spend the 0th vout of the txid 9ca8f9… as its input and assign it to a fresh output that sends value to a fresh address.

Very first, let’s look at the specific output in more detail. We use gettxout to get the details of this unspent output. Transaction outputs are always referenced by txid and vout, and these are the parameters we pass to gettxout :

What we see here is the output that assigned fifty millibits to our address 1hvz… . To spend this output we will create a fresh transaction. Very first, let’s make an address to which we will send the money:

We will send twenty five millibits to the fresh address 1LnfTn… we just created in our wallet. In our fresh transaction, we will spend the fifty millibit output and send twenty five millibits to this fresh address. Because we have to spend the entire output from the previous transaction, we must also generate some switch. We will generate switch back to the 1hvz… address, sending the switch back to the address from which the value originated. Eventually, we will also have to pay a fee for this transaction. To pay the fee, we will reduce the switch output by 0.Five millibits, and come back 24.Five millibits in switch. The difference inbetween the sum of the fresh outputs (25 mBTC + 24.Five mBTC = 49.Five mBTC) and the input (50 mBTC) will be collected as a transaction fee by the miners.

We use createrawtransaction to create this transaction. As parameters to createrawtransaction we provide the transaction input (the fifty millibit unspent output from our confirmed transaction) and the two transaction outputs (money sent to the fresh address and switch sent back to the previous address):

The createrawtransaction instruction produces a raw hex string that encodes the transaction details we supplied. Let’s confirm everything is correct by decoding this raw string using the decoderawtransaction instruction:

That looks correct! Our fresh transaction “consumes” the unspent output from our confirmed transaction and then spends it in two outputs, one for twenty five millibits to our fresh address and one for 24.Five millibits as switch back to the original address. The difference of 0.Five millibits represents the transaction fee and will be credited to the miner who finds the block that includes our transaction.

As you might notice, the transaction contains an empty scriptSig because we haven’t signed it yet. Without a signature, this transaction is meaningless; we haven’t yet proven that we own the address from which the unspent output is sourced. By signing, we eliminate the lock on the output and prove that we own this output and can spend it. We use the signrawtransaction guideline to sign the transaction. It takes the raw transaction hex string as the parameter:

An encrypted wallet must be unlocked before a transaction is signed because signing requires access to the secret keys in the wallet.

The signrawtransaction directive comebacks another hex-encoded raw transaction. We decode it to see what switched, with decoderawtransaction :

Now, the inputs used in the transaction contain a scriptSig , which is a digital signature proving ownership of address 1hvz… and removing the lock on the output so that it can be spent. The signature makes this transaction verifiable by any knot in the bitcoin network.

Now it’s time to submit the freshly created transaction to the network. We do that with the instruction sendrawtransaction , which takes the raw hex string produced by signrawtransaction . This is the same string we just decoded:

The directive sendrawtransaction comebacks a transaction hash (txid) as it submits the transaction on the network. We can now query that transaction ID with gettransaction :

As before, we can also examine this in more detail using the getrawtransaction and decodetransaction directives. These guidelines will comeback the exact same hex string that we produced and decoded previously just before we sent it on the network.

Alternative Clients, Libraries, and Toolkits

Beyond the reference client (bitcoind), other clients and libraries can be used to interact with the bitcoin network and data structures. These are implemented in a multitude of programming languages, suggesting programmers native interfaces in their own language.

Alternative implementations include:

Many more libraries exist in a multitude of other programming languages and more are created all the time.

Libbitcoin and Bitcoin Explorer

The libbitcoin library is a cross-platform C++ development toolkit that supports the libbitcoin-server utter knot and the Bitcoin Explorer (bx) directive line device.

The bx instructions suggest many of the same capabilities as the bitcoind client instructions we illustrated in this chapter. The bx directions also suggest some key management and manipulation devices that are not suggested by bitcoind, including type-2 deterministic keys and mnemonic key encoding, as well as stealth address, payment, and query support.

Installing Bitcoin Explorer

To use Bitcoin Explorer, simply download the signed executable for your operating system. Builds are available for mainnet and testnet for Linux, OS X, and Windows.

Type bx with no parameters to display the list of all available directives (see Appendix D).

Bitcoin Explorer also provides an installer for building from sources on Linux and OS X, as well as Visual Studio projects for Windows. Sources can also be built by hand using Autotools. These also install the libbitcoin library dependency.

Bitcoin Explorer offers many useful directives for encoding and decoding addresses, and converting to and from different formats and representations. Use them to explore the various formats such as Base16 (hex), Base58, Base58Check, Base64, etc.

Installing Libbitcoin

The libbitcoin library provides an installer for building from sources on Linux and OS X, as well as Visual Studio projects for Windows. Sources can also be built by hand using Autotools.

The Bitcoin Explorer installer installs both bx and the libbitcoin library, so if you have built bx from sources, you can skip this step.

pycoin

The Python library pycoin , originally written and maintained by Richard Smooch, is a Python-based library that supports manipulation of bitcoin keys and transactions, even supporting the scripting language enough to decently deal with nonstandard transactions.

The pycoin library supports both Python two (Two.7.x) and Python three (after Trio.Trio), and comes with some handy command-line utilities, ku and tx. To install pycoin 0.42 under Python three in a virtual environment (venv), use the following:

Here’s a sample Python script to fetch and spend some bitcoin using the pycoin library:

For examples using the command-line utilities ku and tx, see Appendix B.

btcd is a full-node bitcoin implementation written in Go. It presently downloads, validates, and serves the blockchain using the exact rules (including bugs) for block acceptance as the reference implementation, bitcoind. It also decently relays freshly mined blocks, maintains a transaction pool, and relays individual transactions that have not yet made it into a block. It ensures that all individual transactions admitted to the pool go after the rules required and also includes the vast majority of the more stringent checks that filter transactions based on miner requirements (“standard” transactions).

One key difference inbetween btcd and bitcoind is that btcd does not include wallet functionality, and this was a very intentional design decision. This means you can’t actually make or receive payments directly with btcd. That functionality is provided by the btcwallet and btcgui projects, which are both under active development. Other notable differences inbetween btcd and bitcoind include btcd support for both HTTP POST requests (such as bitcoind) and the preferred Websockets, and the fact that btcd’s RPC connections are TLS-enabled by default.

Installing btcd

To install btcd for Windows, download and run the msi available at GitHub, or run the following directive on Linux, assuming you already have installed the Go language:

To update btcd to the latest version, just run:

Controlling btcd

btcd has a number of configuration options, which you can view by running:

btcd comes prepackaged with some goodies such as btcctl, which is a command-line utility that can be used to both control and query btcd via RPC. btcd does not enable its RPC server by default; you must configure at minimum both an RPC username and password in the following configuration files:

  • btcd.conf :
  • btcctl.conf :

Or if you want to override the configuration files from the guideline line:

For a list of available options, run the following:

Related video:

Leave a Reply

Your email address will not be published. Required fields are marked *

*