Blockchain: A technical primer
www2.deloitte.comA blockchain is a digital and distributed ledger of transactions, recorded and replicated in real time across a network of computers or nodes (figure 1). Every transaction must be cryptographically validated via a consensus mechanism executed by the nodes before being permanently added as a new “block” at the end of the “chain.” There is no need for a central authority to approve the transaction, which is why blockchain is sometimes referred to as a peer-to-peer trustless mechanism.
I see this kind of explanation a lot. It's misleading at best. A block chain is most definitely not a ledger, as is claimed by too many sources to count.
A block chain is a tool allowing a group of users to produce a single, unique transaction log.
To this end, transactions are grouped into blocks. A block references two pieces of information: (1) its ordered list of transactions; and (2) its parent block. Two or more blocks may claim the same block as a parent. Therefore, a collection of blocks can be assembled into a tree with one or more children pointing to a parent.
Each user maintains its own copy of the tree. All paths through the tree are followed and scored. The path with the highest score is designated as the active chain.
The active chain is what many people think of when they use the term "Blockchain."
However, the structure of this chain is mostly boring and unremarkable. The most interesting thing about it are transactions. Like blocks, transactions link together. Additionally, transactions run programs that induce global state changes.
The secret sauce of "Blockchain" boils down to incentivizing each user to choose the same active chain in an adversarial environment. The best-tested procedure uses "proof-of-work."
If the relative order of events plays an important part in what you're trying to do, and you operate in an adversarial environment, then a block chain might help.
Otherwise, there's not much a block chain can do for you.
I thought a ledger was a transaction log. It was in relation to a ledger I first came by the term transaction (or “trans”).
You might be thinking of a "journal":
Journals and ledgers are where business transactions are recorded in an accounting system. In essence, detail-level information for individual transactions is stored in one of several possible journals, while the information in the journals is then summarized and transferred (or posted) to a ledger. The posting process may take place quite frequently, or could be as infrequent as the end of each reporting period. The information in the ledger is the highest level of information aggregation, from which trial balances and financial statements are produced.
https://www.accountingtools.com/articles/what-is-the-differe...
A ledger can be created from a journal (transaction log), but then again, many other kinds of things can be created from a transaction log (e.g., the "World Computer" idea for Ethereum).
Ah a journal is more like a block. It is transaction with several line items.
This is then transferred into the ledger which is just a list of transactions. Exactly like a blockchain
Why is the distinction between "ledger" and "transaction log" important? Are there technical definitions for the terms in this context that make the terms not interchangeable?
I believe a ledger is the account balances that are computed from a transaction log. So a block on the Bitcoin network is a collection of transactions, whereas a block on the Stellar network is a ledger of the account balances for all "accounts" on the Stellar network.
Each user maintains its own copy of the tree. All paths through the tree are followed and scored. The path with the highest score is designated as the active chain.
The active chain is what many people think of when they use the term "Blockchain."
This is where many people falter when thinking about blockchain. When you tell them that their esoteric blockchain needs to ensure lowest possible latency between peers (nodes), they blank out. They had never heard of the terms fork or the highest amount of work (at least if the chain is secured via PoW).
Well, isn't that what a ledger is, a transaction log?
The "recorded and replicated in real time across a network" thingie really stands out
Blockchain is very applicable to industries where middlemen extract large value by being intermediaries between competing parties.
As an example, in telecom there is the issue of settlement. Every phone call generates a Call Data Record (CDR) that has information about who called who, call length, etc. This is critical because every telco negotiates individual rates for voice, SMS, data roaming, etc. between every other carrier on earth (although often you negotiate with one company that provides a package rate for multiple telcos). Your "unlimited" plan obfuscates the fact that how you use your plan, what carriers the people you call are on, where you use data, reflect different levels of profitability for you as a customer.
Large companies like Amdocs[0] make billions of dollars handling settlement for telcos. Every month, a carrier has to "settle" all of their CDR's with every other carrier. Issues get sorted out because CDR's never match perfectly, and then carriers pay each other to balance out based on their contractual obligations.
A blockchain solution that all telcos agreed on would guarantee data correctness (CDR's all look alike), allow instant settlement and payment (settlement happens once a month), and eliminate the middlemen like Amdocs, Synchronoss, etc. in regards to settlement.
Obviously this is a tough problem to crack, and you'd need to get the telcos on board, but I am explaining this as an example where blockchain could provide real value.
Indeed, Maersk, the largest shipping conglomerate, is going to release a blockchain solution to move the international shipping industry away from paper and to a unified system.[1] Maersk is in a unique position to do this because they are the largest player, and most other companies just follow what they do, but I mention this as an example where blockchain can help competing companies interact more efficiently.
[1] https://www.economist.com/briefing/2018/04/26/the-global-log...
Going from "paper" directly to blockchain, with not a peep about traditional databases.
Surely the executive is just using blockchain as a buzzword for database.
Believe me, I empathize with your comment.
However, I think maybe you're missing one thing that is being proposed as a real thing - and that is, that blockchain has become like a distributed database - I mean, there are people proposing that they can be considered as the same kind of technology, or blockchain tech can be seen as an evolution supporting database technology.
I am still on the fence if I completely believe it, which is probably why I won't get rich building blockchain-based database technology or implementing it. But I think it is a valid consideration and we as technologists should look into what this model (blockchain as a kind of database technology) really represents.
It's another arrow in the architect's quiver, perhaps.
The comment you replied to mentioned that "Large companies ... make billions of dollars handling [databases] for $companies." Central databases have become excuses to erect toll booths that blockchain may eliminate.
So why doesn't $company roll their own database? Why go the extra mile and roll out an infamously-inefficient blockchain?
It's a lot of work, that's why.
Because $company is only one of a number of competitors and none of them is going to let one of them handle the pot.
And because there's distribution but also only a small number of semi-trusted peers, a "blockchain" would be efficient as it wouldn't have to be a proof of work, it would only need to be a signed "public" ledger (public in the sense that all companies can access and add to it). Don't think bitcoin, think a git repo with a bunch of committer signing all their commits.
This stuff is being proposed for data that needs consensus between multiple companies, so each company can't have their own.
If you don't trust your business partner, don't do business with them. If there are problems, contracts and courts decide what to do. There is no need for blockchain.
This is not how the real world works.
When you have large industries with thousands of peering organizations all over the world, you can't just "trust" -- there is, in fact, no metric for trust and no ability to measure trust. That's why the middlemen exist.
This is especially true in finance, healthcare, and other industries where blockchains are being used.
How do you make a traditional database immutable? Who controls writes and updates? Well it’s a matter of consensus.
Blockchains are not immutable - the DAO hack and subsequent hard-fork proved that to the tune of millions of dollars.
That was a fork, not a mutation of the database. You can still today go back and see the history play out in the different chains.
Who determines if competing companies are entering authentic data into the communal blockchain? How do you address a consensus attack in an aggressively competitive industry with relatively few players?
Immutable garbage is still garbage.
Databases are perfectly compatible with any variety of authentication layers.
You are describing a Master Data Management(MDM) system. Also referred to as "single source of truth" in a data oriented system. The data in a MDM is homogeneous. The most important requisite for MDM's effectiveness that every downstream system needs to get data from MDM. No system should generate it's own data.
While those are general rule in practice, it is a difficult task to pull off. The problem is not technical but rather due to people. There are tons of department in a company. And every time an input is made things change a little. It's due to human mistakes, software restrictions and lot more.
I haven't worked in teleco but here's an example on what goes wrong when data is recorded. Marketing will send out flyers to say 100 people. One of them is named ABC XYZ. This person is interested in a phone line from say AT&T.
When sales of AT&T take this up due to some problem, the name is recorded as ABC-XYZ ie with a hyphen. Now this data is human readable but doesn't make sense to a computer. So, when it comes to tallying up marketing and sales it doesn't work.
Now things get worse when a connection has been sold, the address field in sales might be smaller compared to a full fledged customer system. So, sales says address is "XYZ Street" whereas customer system says "SSS Boulervad, XYZ Street". And then this snowballs further.
To resolve this many companies try to implement a UCM - Unified Customer Management systems, a branch of MDM. But then there are people's ego, money considerations etc come in the way. So companies don't bother with it and go with settlement vendors.
Now, this is a similar situation when it comes to CDR. Different equipment means different sets of data, that's why they need a settlement company.
So, the problem is not a simple - middlemen eating money. There are lot of things going on.
> CDR's never match perfectly
With a blockchain based system, what happens if two telcos send CDRs that don't match?
Typically a conflict-resolution mechanism will be built into the consensus mechanism, with the rules set by the peers.
But there will always be transparency (even the bad CDRs will be logged) and accountability (you know who said what, so the incentive for fraud is significantly decreased.)
But if all the telcos have to agree anyway, why not just use a centralized db with cryptographic signatures?
A key element of blockchain is the associated business logic (smart contracts). You need an immutable ledger and you need agreed upon business logic.
You have your choice of consensus algorithms. Most of the time when people say blockchain, they mean the public networks like bitcoin or ethereum. But if you move to a private, permissioned network model, you can use consensus algorithms that don’t need mining and have very high transaction speeds. This is the approach taken by IBM Blockchain (via the Linux Foundation’s open source hyperledger fabric project) and a few others.
So eg. if telco A says a call was 8 minutes long and telco B says it was 15 minutes - what kind of smart contract / business logic decides how long the call was?
Resolving these issues can happen in several ways.
Since all participants are operating with the same data store, you can flag these conflicts in real time. You might even have individual transactions to mark start and end times, such that conflicts like this become impossible.
You might escalate to human resolution. IBM Global Finance implemented a blockchain system for thousands of its business partners in order to help resolve invoicing and payment issues. By having a trusted single source of truth, the dispute resolution process was dramatically accelerated. I don’t recall the exact numbers, but they went from something like 90 days average resolution time to just 1-3 days. This resulted in reducing working capital needs by about ~$100m, thus saving a few million dollars per year just in interest charges.
I can see the benefit of real time operation, and I can see the benefit of using a single, standardized invoicing / payment system. I don't understand why you need a blockchain for any of that.
In scenarios where there exists a trusted central party, blockchain is not an applicable solution. Transitioning from such a centralized system to blockchain would require substantial investment for hypothetical gains.
However there are many scenarios where it would be infeasible to have a central party. A farmer grows an organic coffee bean in one country and it needs to get to a coffee shop in another country. There are multiple parties and industries involved: the farmer's supplier, the organic certification authorities, the trucking companies that transport the product between farmer/roaster/port/coffee shop, the foreign customs agency, the ocean shipping company, the numerous insurance companies involved, the domestic customs agency, the wholesale distributors, the coffee shops. Throw in the IoT sensors for measuring watering, pesticide use, transport climate, storage climate, and so on that report to the blockchain (which records out-of-tolerance conditions).
Let's say the consumer wants proof the product is organic, how would you do it? What if the beans arrive spoiled, who is responsible? How about if there is an accident, which of the insurance companies is responsible? Who do they pay? What if an outbreak of disease results from the coffee, how would you find the culprit?
The thing is, the world today works. We have records on paper, databases, and huge numbers of people involved in figuring out those questions (though some questions and business models are effectively impossible, like "What if the farmer wants to set the price for a cup of coffee, instead of selling to distributors, how would they do that?").
But these people don't talk to each other. What if there was a trusted business/industrial network of databases that could be integrated into business operations? What new business models, efficiencies, and products would that system enable? That is the promise of blockchain. Bitcoin and cryptocurrencies are silly toys by comparison.
Blockchain is in its infancy. By analogy, if blockchain were the internet, it would be at the stage of dialing into your local BBS by using an acoustic coupler. It is hard to imagine what the world will look like in 30-40 years when industrial/business uses of blockchain are advanced and pervasive.
That's conflict-resolution logic, and up to the peers to decide a priori (they will likely already have this, since it is not an uncommon scenario.)
The value here (in addition to decentralized validation and consensus) is also in transparency and accountability, which disincentivizes fraud.
> That's conflict-resolution logic, and up to the peers to decide a priori (they will likely already have this, since it is not an uncommon scenario.)
Or you could just say you don't know?
> Or you could just say you don't know?
Ugh, there's really no need for that! I actually worked in telecom for years, including integrating cellular network gear with third-party billing and settlement systems (like Amdocs.)
Billing is hugely complicated and the third party systems cost in the millions of dollars in licensing and maintenance fees. They include thousands of rules to deal with the variety of scenarios (especially in conflict-resolution) that play out when peers settle with each other. The peers themselves work out large and complex legal contracts before they're encoded into these systems.
I wasn't saying you don't know because you're ignorant about the telecoms industry. I think you don't know how to algorithmically resolve a conflict between two adversaries who assert significantly different versions of history, because nobody knows how to do that. I don't have your telecoms insider knowledge, but I feel fairly confident I know what would happen if a telco was suspected of inexcusable misreporting: people would check the data, have meetings and decide what they wanted to do about it. Maybe there's scope for automation of a bunch of deterministic stuff that would happen before it got to that point, but the actual resolution of actual conflict has to be a human process.
> I wasn't saying you don't know because you're ignorant about the telecoms industry. I think you don't know how to algorithmically resolve a conflict between two adversaries who assert significantly different versions of history, because nobody knows how to do that.
Yes, this is why they negotiate this stuff before the fact. Switches see different things all the time, especially when there are bugs, outages, or misconfigurations.
The lawyers come in when there is actual conflict, e.g., when fraud is suspected, or anything outside the terms of what was already negotiated.
The value of a blockchain here is that these rules (the prenegotiated ones) can be executed by verifiable smart contracts, instead of trusting a third party like Amdocs. If there is a real conflict, there will still be lawyers involved, but with the added advantage of an immutable public (to the federation) log. The bonus (which is yet to be proven) is that costs will be driven down significantly without the middleman.
How is a private blockchain functionally different from the average database?
IBM can charge 8-figure consulting fees instead of 7.
The database is distributed among independent nodes.
Each participant has their own copy of the immutable ledger and world state.
Entries are cryptographically signed.
All interactions run through smart contracts (you can’t just insert/update).
You can form channels to engage in private transactions between a subset of the network, without divulging the existence of the transaction to the full network.
Because then you have to decide who you are going to trust to run the database.
That's not really a hard problem though, .e.g. git and github; any changes made on the host are easily and immediately detectable and verifiable, no block chain necessary.
So I am assuming there is no proof of work involved in this system, correct? If so then it sort of escapes the normal criticisms of blockchain tech (extremely slow and expensive to run)
They are going to end up with a big database.
Also, let me know when we can fire our ops (finanace) and make billions more. Hope you guys appreciate how huge this would be, billions are spent daily to recon trades and positions. You mean to tell me some guy already figured it all out and all we have to do is write a wee bit of code.
Next we are going to get the Hasidic Jews on 5th ave to start using spreadsheets instead of little notebooks so they can price fix diamonds and settings faster.
> In fact, it can be considered one of the biggest technology breakthroughs in recent history, similar to the advent of the Internet in the early 1990s.
If this were the case, wouldn't we see the technology in the wild more? That seems like a bold claim for something that is still being vetted and explored.
I certainly see value in blockchain but the applications that are using it today don't excite me. The hype puts me off but I am hoping to see it bloom into other areas of software that aren't ICOs or proof of concepts.
big changes usually require more time in the adoption phases..autos did at the beginning of the 19th century.
Technologies that are actually useful tend to be aggressively adopted by military, academia, industry. Internet and IC engine were.
On the contrary, blockchain finds its activism among SQL-illiterate people, and certainly not the aforementioned groups. Mind that we're already a decade in.
What do you define as aggressively adopted? The Ethereum Enterprise Alliance has 500+ members including many Fortune 500 companies.
There is widespread experimention with blockchain technology by government, academia and industry.
For example, the majority of central banks are experimenting with blockchain/distributed-ledger technology:
https://qz.com/1083712/one-in-five-central-banks-say-they-wi...
NASA is researching using blockchain technology for spacecraft navigation:
https://www.uakron.edu/engineering/ECE/news-detail.dot?newsI...
With a SQL database, how do you achieve consensus on a proposed transaction without a middleman taking an arbitrary cut of the proceeds?
The two parties either agree, or they don't. Full stop. Blockchain, or any buzzword technology, doesn't solve this eternal problem.
Third parties have historically been pretty good at sussing out consensus from the two. Judicial systems, market exchanges, etc.
When you pay a middleman, you are literally paying for Trust-as-a-Service, among other things (brokerage, etc).
Blockchain also costs money to build and maintain as a service, but it's probably the fact that the enormous cost is socialized that makes it an easy pill for many to swallow.
Who maintains the source of truth for this record of agreement? If it is one entity, that's the unaccountable middleman. Blockchain is not right for every environment, but there are industries where the middlemen are taking more than the equivalent of every party maintaining a blockchain team.
And court cases are even more expensive.
> With a SQL database, how do you achieve consensus on a proposed transaction without a middleman taking an arbitrary cut of the proceeds?
Um, there is still a middleman taking a cut with Satoshi's Glorious Blockchain. It's called transaction fees and they go to the miners who piss away small countries worth of electricity in order to validate a pathetically small number of transactions. You didn't think proof of work was free, right?
No reasonable industry would use Bitcoin's proof of work to transact between <10 semi-trustworthy parties.
We're talking about private, permissioned blockchains.
> permissioned
I.e. "trusted". So what does this bring over existing consensus algorithms, besides the "blockchain" buzzword hype?
Which ones are you thinking of? They should support immutability and auditability as first-class citizens while being resistant to third-party manipulation/coercion.
All consensus does in bitcoin is determine whether a transaction is possible or impossible (avoid double spends).
With a SQL database, this is not an issue because SQL transactions can have constraints preventing said double spend.That is to say, the SQL database can take a lock on the row and ensure no other concurrent transaction modifies it in parallel.
Consensus solves a problem that only exists in a decentralized system.
Since SQL databases are centralized, that problem doesn't exist. Even for multi-master / distributed SQL variants, there's typically a way to wait for a write to commit across N nodes before going on, which solves the problem much more easily.
In short, I think your question makes no sense.
Not to mention that the industries that could be most transformed by adopting blockchains -- banking being the obvious one -- are among the most slow-moving and change-averse.
I find this to be less of a whitepaper and more of an opinion piece. For e.g.
"For example, two business parties transacting would not need to maintain their own record of the transaction and would use the blockchain as a single source of truth instead."
Ummmmmm...isn't the whole idea of blockchain that we all have a copy (our version of the truth) and a transaction is validated by several participants or miners? That's basically how we get rid of the middlemen? The sentence is clearly trying to convey that this is somehow more efficient than a current system based on a centralized authority, but there's no actual evidence that this is true.
Blockchain might bring efficiency to an industry like shipping that currently uses a VERY inefficient system, but it's disingenuous to tell others (especially if you are trying to sell something), that it is more efficient than a well implemented centralized system.
They explain the basics of permisionless blockchains and they go on and say "Prominent blockchain consortia include Enterprise Ethereum Alliance, R3, and Hyperledger", this can confuse people since R3 and Hyperledger are quite different from a permissionless chain with mining, and the value added of such platforms without crypto-currencies is highly debatable in my opinion.
> In fact, it can be considered one of the biggest technology breakthroughs in recent history, similar to the advent of the Internet in the early 1990s.
Audibly sighing. read this: https://shkspr.mobi/blog/2018/06/how-i-became-leonardo-da-vi...
It's the internet of shit in crypto, all over again. Come on. There's nothing like 50% of the swarm simply declaring your transactions null and void, but that's obviously how the blockchain is currently operating for some ... uh, criminal reason.
"technical primer", yeah right.
I guess this counts as "technical" to people at Deloitte.
Pretty good primer for newcomers, though it seems to take it as a given that blockchains are (or to hear them say it, "blockchain is") just fully as amazing and applicable as the hype suggests, and here's what you are going to have to do when you inevitably get with the program!
This is understandable, given the fact that this is from Deloitte, who would like to sell consulting services.
Another amateur's writing about blockchain. If you want the immutability, the blockchain is not immutable, just difficult to modify. Also you only need hash signature for this purpose.