You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Jörn Franke 8affcdd053 bump up log4j2 9 months ago
project bump up sbt 9 months ago
src bump up versions 2 years ago
.gitignore fix #21 10 months ago
.travis.yml update scala 2.12 4 years ago
CONTRIBUTING.md Create CONTRIBUTING.md 5 years ago
LICENSE Initial 6 years ago
NOTICE Notice file 6 years ago
README.md bump up versions 10 months ago
SECURITY.md Update SECURITY.md 5 years ago
build.sbt bump up log4j2 9 months ago

README.md

spark-hadoopcryptoledger-ds

Build Status Codacy Badge

A Spark datasource for the HadoopCryptoLedger library.

Currently this datasource supports the following formats of the HadoopCryptoLedger library (see schemas at the end of the page):

  • Bitcoin and Altcoin Blockchain
    • Bitcoin Block Datasource format: org.zuinnote.spark.bitcoin.block
    • Bitcoin Transaction Datasource format: org.zuinnote.spark.bitcoin.transaction
  • Ethereum and Altcoin Blockchain
    • Ethereum Block Datasource format: org.zuinnote.spark.ethereum.block

This datasource is available on Spark-packages.org and on Maven Central.

Find here the status from the Continuous Integration service: https://travis-ci.org/ZuInnoTe/spark-hadoopcryptoledger-ds/

Release Notes

Find the latest release information here

Options

The following options are mapped to the following options of the HadoopCryptoLedger library (Explanation):

  • Bitcoin and Altcoins
    • "magic" is mapped to "hadoopcryptoledger.bitcoinblockinputformat.filter.magic"
    • "maxblockSize" is mapped to "hadoopcryptoledger.bitcoinblockinputformat.maxblocksize"
    • "useDirectBuffer" is mapped to "hadoopcryptoledeger.bitcoinblockinputformat.usedirectbuffer"
    • "isSplittable" is mapped to "hadoopcryptoledeger.bitcoinblockinputformat.issplitable"
    • "readAuxPOW" is mapped to "hadoopcryptoledeger.bitcoinblockinputformat.readauxpow"
    • "enrich" adds the transaction hash to each transaction in the Bitcoin block
  • Ethereum and Altcoins
    • "maxblockSize" is mapped to "hadoopcryptoledger.ethereumlockinputformat.maxblocksize"
    • "useDirectBuffer" is mapped to "hadoopcryptoledeger.ethereumblockinputformat.usedirectbuffer"
    • "enrich" in case of true it additional data is calculated for transactions: sendAddress and hash. Default: false. Note: you must include the bouncycastle dependency to use this.
    • "chainId" this is needed if you enrich to correctly calculate the SendAddress. Examples are 1 Frontier (Mainnet), 2 Morden, 3 Ropsten, but other chainIds exist for Altcoins based on Ethereum. Default 1.

Setup

libraryDependencies += "com.github.zuinnote" %% "spark-hadoopcryptoledger-ds" % "1.3.1"

The library is currently published for 2.11 and 2.12. Scala 2.12 requires that you use at least Spark 2.4.0

If you use Ethereum and the enrich functionality then you need the Bouncycastle dependency:

libraryDependencies += "org.bouncycastle" %% "sbcprov-ext-jdk15on" % "1.64"

Information Spark 2.0 and 2.1 BigInteger too large

Spark 2.0 and Spark 2.1 have a bug (see https://issues.apache.org/jira/browse/SPARK-20341) that does not allow Big Integers with a precision large than 19. You may face those in the Ethereum blockchain. We recommend to use Spark 2.3 latest.

Information Spark 2.2 and outdated Bouncy Castle library

As omervk and liorregev point out, Spark 2.2 uses jets3t 0.9.3, which depends on an outdated version of Bouncy Castle. Unfortunately, this outdated version does not support the cryptographic operations needed for enrichment of Ethereum data (SendAddress and TransactionHash).

You have the following alternatives:

  1. Shade the latest BC library in your application (hadoopcryptoledger has it only as a provided dependency)
  2. Remove the outdated bouncy castle libraries from the Spark libraries (and do not use jets3t); or
  3. Wait for an updated jets3t version, which then need to be included into an updated Spark version

Other Spark versions do not currently show any issues (Spark 1.6, 2.0, 2.1 or 2.3 are not affected).

Develop

The following sections describe some example code.

Scala

Bitcoin and Altcoins

This example loads Bitcoin Blockchain data from the folder "/user/bitcoin/input" using the BitcoinBlock representation (format).

val sqlContext = new SQLContext(sc)
val df = sqlContext.read
   .format("org.zuinnote.spark.bitcoin.block")
   .option("magic", "F9BEB4D9")
   .load("/user/bitcoin/input")

// Or if you'd like a Dataset version (Spark 2.0+)...

import org.zuinnote.spark.bitcoin.model._
import df.sparkSession.implicits._

// Also available: EnrichedBitcoinBlock, BitcoinBlockWithAuxPOW, EnrichedBitcoinBlockWithAuxPOW
val ds: Dataset[BitcoinBlock] = df.as[BitcoinBlock]

The HadoopCryptoLedger library provides an example for scala using the data source library: https://github.com/ZuInnoTe/hadoopcryptoledger/wiki/Use-HadoopCrytoLedger-library-as-Spark-DataSource

Ethereum and Altcoins

This example loads Ethereum Blockchain data from the folder "/user/ethereum/input" using the EthereumBlock representation (format).

val sqlContext = new SQLContext(sc)
val df = sqlContext.read
   .format("org.zuinnote.spark.ethereum.block")
   .option("enrich", "false")
   .load("/user/ethereum/input")

// Or if you'd like a Dataset version (Spark 2.0+)...

import org.zuinnote.spark.ethereum.model._
import df.sparkSession.implicits._

// Also available: EnrichedEthereumBlock
val ds: Dataset[EthereumBlock] = df.as[EthereumBlock]

The HadoopCryptoLedger library provides an example for scala using the data source library: ledger/wiki/Use-HadoopCrytoLedger-library-as-Spark-datasource-to-read-Ethereum-data

Java

Bitcoin and Altcoins

This example loads Bitcoin Blockchain data from the folder "/user/bitcoin/input" using the BitcoinBlock representation (format).

import org.apache.spark.sql.SQLContext

SQLContext sqlContext = new SQLContext(sc);
DataFrame df = sqlContext.read()
   .format("org.zuinnote.spark.bitcoin.block")
   .option("magic", "F9BEB4D9")
   .load("/user/bitcoin/input");

Ethereum and Altcoins

This example loads Ethereum Blockchain data from the folder "/user/ethereum/input" using the EthereumBlock representation (format).

import org.apache.spark.sql.SQLContext

SQLContext sqlContext = new SQLContext(sc);
DataFrame df = sqlContext.read()
   .format("org.zuinnote.spark.ethereum.block")
   .option("enrich", "false")
   .load("/user/ethereum/input");

R

Bitcoin and Altcoins

This example loads Bitcoin Blockchain data from the folder "/user/bitcoin/input" using the BitcoinBlock representation (format).

library(SparkR)

Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.github.zuinnote:spark-hadoopcrytoledger-ds_2.11:1.3.1" "sparkr-shell"')
sqlContext <- sparkRSQL.init(sc)

df <- read.df(sqlContext, "/user/bitcoin/input", source = "org.zuinnote.spark.bitcoin.block", magic = "F9BEB4D9")

Ethereum and Altcoins

This example loads Ethereum Blockchain data from the folder "/user/ethereum/input" using the EthereumBlock representation (format).

library(SparkR)

Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.github.zuinnote:spark-hadoopcrytoledger-ds_2.11:1.3.1" "sparkr-shell"')
sqlContext <- sparkRSQL.init(sc)

df <- read.df(sqlContext, "/user/ethereum/input", source = "org.zuinnote.spark.ethereum.block", enrich = "false")

Python

Bitcoin and Altcoins

This example loads Bitcoin Blockchain data from the folder "/user/bitcoin/input" using the BitcoinBlock representation (format).

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

df = sqlContext.read.format('org.zuinnote.spark.bitcoin.block').options(magic='F9BEB4D9').load('/user/bitcoin/input')

Ethereum and Altcoins

This example loads Ethereum Blockchain data from the folder "/user/ethereum/input" using the EthereumBlock representation (format).

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

df = sqlContext.read.format('org.zuinnote.spark.ethereum.block').options(enrich='false').load('/user/ethereum/input')

SQL

Bitcoin and Altcoins

The following statement creates a table that contains Bitcoin Blockchain data in the folder /user/bitcoin/input

CREATE TABLE BitcoinBlockchain
USING  org.zuinnote.spark.bitcoin.block
OPTIONS (path "/user/bitcoin/input", magic "F9BEB4D9")

Ethereum and Altcoins

The following statement creates a table that contains Ethereum Blockchain data in the folder /user/ethereum/input

CREATE TABLE EthereumBlockchain
USING  org.zuinnote.spark.ethereum.block
OPTIONS (path "/user/ethereum/input", enrich "false")

Schemas

Format: org.zuinnote.spark.bitcoin.block (readAuxPOW=false,enrich=true)

root
 |-- blockSize: long (nullable = false)
 |-- magicNo: binary (nullable = false)
 |-- version: long (nullable = false)
 |-- time: long (nullable = false)
 |-- bits: binary (nullable = false)
 |-- nonce: long (nullable = false)
 |-- transactionCounter: long (nullable = false)
 |-- hashPrevBlock: binary (nullable = false)
 |-- hashMerkleRoot: binary (nullable = false)
 |-- transactions: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- version: long (nullable = false)
 |    |    |-- marker: byte (nullable = false)
 |    |    |-- flag: byte (nullable = false)
 |    |    |-- inCounter: binary (nullable = false)
 |    |    |-- outCounter: binary (nullable = false)
 |    |    |-- listOfInputs: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- prevTransactionHash: binary (nullable = false)
 |    |    |    |    |-- previousTxOutIndex: long (nullable = false)
 |    |    |    |    |-- txInScriptLength: binary (nullable = false)
 |    |    |    |    |-- txInScript: binary (nullable = false)
 |    |    |    |    |-- seqNo: long (nullable = false)
 |    |    |-- listOfOutputs: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- value: decimal(38,0) (nullable = false)
 |    |    |    |    |-- txOutScriptLength: binary (nullable = false)
 |    |    |    |    |-- txOutScript: binary (nullable = false)
 |    |    |-- listOfScriptWitnessItem: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = false)
 |    |    |    |    |-- stackItemCounter: binary (nullable = false)
 |    |    |    |    |-- scriptWitnessList: array (nullable = true)
 |    |    |    |    |    |-- element: struct (containsNull = false)
 |    |    |    |    |    |    |-- witnessScriptLength: binary (nullable = false)
 |    |    |    |    |    |    |-- witnessScript: binary (nullable = false)
 |    |    |-- lockTime: long (nullable = false)
 |    |    |-- currentTransactionHash: binary (nullable = false)

Format: org.zuinnote.spark.bitcoin.block (readAuxPOW=false,enrich=false)

root
 |-- blockSize: long (nullable = false)
 |-- magicNo: binary (nullable = false)
 |-- version: long (nullable = false)
 |-- time: long (nullable = false)
 |-- bits: binary (nullable = false)
 |-- nonce: long (nullable = false)
 |-- transactionCounter: long (nullable = false)
 |-- hashPrevBlock: binary (nullable = false)
 |-- hashMerkleRoot: binary (nullable = false)
 |-- transactions: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- version: long (nullable = false)
 |    |    |-- marker: byte (nullable = false)
 |    |    |-- flag: byte (nullable = false)
 |    |    |-- inCounter: binary (nullable = false)
 |    |    |-- outCounter: binary (nullable = false)
 |    |    |-- listOfInputs: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- prevTransactionHash: binary (nullable = false)
 |    |    |    |    |-- previousTxOutIndex: long (nullable = false)
 |    |    |    |    |-- txInScriptLength: binary (nullable = false)
 |    |    |    |    |-- txInScript: binary (nullable = false)
 |    |    |    |    |-- seqNo: long (nullable = false)
 |    |    |-- listOfOutputs: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- value: decimal(38,0) (nullable = false)
 |    |    |    |    |-- txOutScriptLength: binary (nullable = false)
 |    |    |    |    |-- txOutScript: binary (nullable = false)
 |    |    |-- listOfScriptWitnessItem: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = false)
 |    |    |    |    |-- stackItemCounter: binary (nullable = false)
 |    |    |    |    |-- scriptWitnessList: array (nullable = true)
 |    |    |    |    |    |-- element: struct (containsNull = false)
 |    |    |    |    |    |    |-- witnessScriptLength: binary (nullable = false)
 |    |    |    |    |    |    |-- witnessScript: binary (nullable = false)
 |    |    |-- lockTime: long (nullable = false)
 

Format: org.zuinnote.spark.bitcoin.block (readAuxPOW=true,enrich=true)

root
 |-- blockSize: long (nullable = false)
 |-- magicNo: binary (nullable = false)
 |-- version: long (nullable = false)
 |-- time: long (nullable = false)
 |-- bits: binary (nullable = false)
 |-- nonce: integer (nullable = false)
 |-- transactionCounter: long (nullable = false)
 |-- hashPrevBlock: binary (nullable = false)
 |-- hashMerkleRoot: binary (nullable = false)
 |-- transactions: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- version: long (nullable = false)
 |    |    |-- marker: byte (nullable = false)
 |    |    |-- flag: byte (nullable = false)
 |    |    |-- inCounter: binary (nullable = false)
 |    |    |-- outCounter: binary (nullable = false)
 |    |    |-- listOfInputs: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- prevTransactionHash: binary (nullable = false)
 |    |    |    |    |-- previousTxOutIndex: long (nullable = false)
 |    |    |    |    |-- txInScriptLength: binary (nullable = false)
 |    |    |    |    |-- txInScript: binary (nullable = false)
 |    |    |    |    |-- seqNo: long (nullable = false)
 |    |    |-- listOfOutputs: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- value: decimal(38,0) (nullable = false)
 |    |    |    |    |-- txOutScriptLength: binary (nullable = false)
 |    |    |    |    |-- txOutScript: binary (nullable = false)
 |    |    |-- listOfScriptWitnessItem: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = false)
 |    |    |    |    |-- stackItemCounter: binary (nullable = false)
 |    |    |    |    |-- scriptWitnessList: array (nullable = true)
 |    |    |    |    |    |-- element: struct (containsNull = false)
 |    |    |    |    |    |    |-- witnessScriptLength: binary (nullable = false)
 |    |    |    |    |    |    |-- witnessScript: binary (nullable = false)
 |    |    |-- lockTime: long (nullable = false)
 |    |    |-- currentTransactionHash: binary (nullable = false)
 |-- auxPOW: struct (nullable = true)
 |    |-- version: long (nullable = false)
 |    |-- coinbaseTransaction: struct (nullable = false)
 |    |    |-- version: long (nullable = false)
 |    |    |-- inCounter: binary (nullable = false)
 |    |    |-- outCounter: binary (nullable = false)
 |    |    |-- listOfInputs: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- prevTransactionHash: binary (nullable = false)
 |    |    |    |    |-- previousTxOutIndex: long (nullable = false)
 |    |    |    |    |-- txInScriptLength: binary (nullable = false)
 |    |    |    |    |-- txInScript: binary (nullable = false)
 |    |    |    |    |-- seqNo: long (nullable = false)
 |    |    |-- listOfOutputs: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- value: decimal(38,0) (nullable = false)
 |    |    |    |    |-- txOutScriptLength: binary (nullable = false)
 |    |    |    |    |-- txOutScript: binary (nullable = false)
 |    |    |-- lockTime: long (nullable = false)
 |    |-- parentBlockHeaderHash: binary (nullable = false)
 |    |-- coinbaseBranch: struct (nullable = false)
 |    |    |-- numberOfLinks: binary (nullable = false)
 |    |    |-- links: array (nullable = false)
 |    |    |    |-- element: binary (containsNull = true)
 |    |    |-- branchSideBitmask: binary (nullable = false)
 |    |-- auxBlockChainBranch: struct (nullable = false)
 |    |    |-- numberOfLinks: binary (nullable = false)
 |    |    |-- links: array (nullable = false)
 |    |    |    |-- element: binary (containsNull = true)
 |    |    |-- branchSideBitmask: binary (nullable = false)
 |    |-- parentBlockHeader: struct (nullable = false)
 |    |    |-- version: long (nullable = false)
 |    |    |-- previousBlockHash: binary (nullable = false)
 |    |    |-- merkleRoot: binary (nullable = false)
 |    |    |-- time: long (nullable = false)
 |    |    |-- bits: binary (nullable = false)
 |    |    |-- nonce: long (nullable = false)

Format: org.zuinnote.spark.bitcoin.block (readAuxPOW=true,enrich=false)

root
 |-- blockSize: long (nullable = false)
 |-- magicNo: binary (nullable = true)
 |-- version: long (nullable = false)
 |-- time: long (nullable = false)
 |-- bits: binary (nullable = true)
 |-- nonce: long (nullable = false)
 |-- transactionCounter: long (nullable = false)
 |-- hashPrevBlock: binary (nullable = true)
 |-- hashMerkleRoot: binary (nullable = true)
 |-- transactions: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- version: long (nullable = false)
 |    |    |-- marker: byte (nullable = false)
 |    |    |-- flag: byte (nullable = false)
 |    |    |-- inCounter: binary (nullable = true)
 |    |    |-- outCounter: binary (nullable = true)
 |    |    |-- listOfInputs: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- prevTransactionHash: binary (nullable = true)
 |    |    |    |    |-- previousTxOutIndex: long (nullable = false)
 |    |    |    |    |-- txInScriptLength: binary (nullable = true)
 |    |    |    |    |-- txInScript: binary (nullable = true)
 |    |    |    |    |-- seqNo: long (nullable = false)
 |    |    |-- listOfOutputs: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- value: decimal(38,0) (nullable = true)
 |    |    |    |    |-- txOutScriptLength: binary (nullable = true)
 |    |    |    |    |-- txOutScript: binary (nullable = true)
 |    |    |-- listOfScriptWitnessItem: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- stackItemCounter: binary (nullable = true)
 |    |    |    |    |-- scriptWitnessList: array (nullable = true)
 |    |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |    |-- witnessScriptLength: binary (nullable = true)
 |    |    |    |    |    |    |-- witnessScript: binary (nullable = true)
 |    |    |-- lockTime: long (nullable = false)
 |-- auxPOW: struct (nullable = true)
 |    |-- version: long (nullable = false)
 |    |-- coinbaseTransaction: struct (nullable = true)
 |    |    |-- version: long (nullable = false)
 |    |    |-- inCounter: binary (nullable = true)
 |    |    |-- outCounter: binary (nullable = true)
 |    |    |-- listOfInputs: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- prevTransactionHash: binary (nullable = true)
 |    |    |    |    |-- previousTxOutIndex: long (nullable = false)
 |    |    |    |    |-- txInScriptLength: binary (nullable = true)
 |    |    |    |    |-- txInScript: binary (nullable = true)
 |    |    |    |    |-- seqNo: long (nullable = false)
 |    |    |-- listOfOutputs: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- value: decimal(38,0) (nullable = true)
 |    |    |    |    |-- txOutScriptLength: binary (nullable = true)
 |    |    |    |    |-- txOutScript: binary (nullable = true)
 |    |    |-- lockTime: long (nullable = false)
 |    |-- parentBlockHeaderHash: binary (nullable = true)
 |    |-- coinbaseBranch: struct (nullable = true)
 |    |    |-- numberOfLinks: binary (nullable = true)
 |    |    |-- links: array (nullable = true)
 |    |    |    |-- element: binary (containsNull = true)
 |    |    |-- branchSideBitmask: binary (nullable = true)
 |    |-- auxBlockChainBranch: struct (nullable = true)
 |    |    |-- numberOfLinks: binary (nullable = true)
 |    |    |-- links: array (nullable = true)
 |    |    |    |-- element: binary (containsNull = true)
 |    |    |-- branchSideBitmask: binary (nullable = true)
 |    |-- parentBlockHeader: struct (nullable = true)
 |    |    |-- version: long (nullable = false)
 |    |    |-- previousBlockHash: binary (nullable = true)
 |    |    |-- merkleRoot: binary (nullable = true)
 |    |    |-- time: long (nullable = false)
 |    |    |-- bits: binary (nullable = true)
 |    |    |-- nonce: long (nullable = false)

Format: org.zuinnote.spark.bitcoin.transaction

root
 |-- currentTransactionHash: binary (nullable = true)
 |-- version: long (nullable = false)
 |-- marker: byte (nullable = false)
 |-- flag: byte (nullable = false)
 |-- inCounter: binary (nullable = true)
 |-- outCounter: binary (nullable = true)
 |-- listOfInputs: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- prevTransactionHash: binary (nullable = true)
 |    |    |-- previousTxOutIndex: long (nullable = false)
 |    |    |-- txInScriptLength: binary (nullable = true)
 |    |    |-- txInScript: binary (nullable = true)
 |    |    |-- seqNo: long (nullable = false)
 |-- listOfOutputs: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- value: decimal(38,0) (nullable = true)
 |    |    |-- txOutScriptLength: binary (nullable = true)
 |    |    |-- txOutScript: binary (nullable = true)
 |-- listOfScriptWitnessItem: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- stackItemCounter: binary (nullable = true)
 |    |    |-- scriptWitnessList: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- witnessScriptLength: binary (nullable = true)
 |    |    |    |    |-- witnessScript: binary (nullable = true)
 |-- lockTime: long (nullable = false)
                                                                                                                                                       

Format: org.zuinnote.spark.ethereum.block (enrich=true)

root
 |-- ethereumBlockHeader: struct (nullable = true)
 |    |-- parentHash: binary (nullable = true)
 |    |-- uncleHash: binary (nullable = true)
 |    |-- coinBase: binary (nullable = true)
 |    |-- stateRoot: binary (nullable = true)
 |    |-- txTrieRoot: binary (nullable = true)
 |    |-- receiptTrieRoot: binary (nullable = true)
 |    |-- logsBloom: binary (nullable = true)
 |    |-- difficulty: binary (nullable = true)
 |    |-- timestamp: long (nullable = false)
 |    |-- number: decimal(38,0) (nullable = true)
 |    |-- numberRaw: binary (nullable = true)
 |    |-- gasLimit: decimal(38,0) (nullable = true)
 |    |-- gasLimitRaw: binary (nullable = true)
 |    |-- gasUsed: decimal(38,0) (nullable = true)
 |    |-- gasUsedRaw: binary (nullable = true)
 |    |-- mixHash: binary (nullable = true)
 |    |-- extraData: binary (nullable = true)
 |    |-- nonce: binary (nullable = true)
 |-- ethereumTransactions: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- nonce: binary (nullable = true)
 |    |    |-- value: decimal(38,0) (nullable = true)
 |    |    |-- valueRaw: binary (nullable = true)
 |    |    |-- receiveAddress: binary (nullable = true)
 |    |    |-- gasPrice: decimal(38,0) (nullable = true)
 |    |    |-- gasPriceRaw: binary (nullable = true)
 |    |    |-- gasLimit: decimal(38,0) (nullable = true)
 |    |    |-- gasLimitRaw: binary (nullable = true)
 |    |    |-- data: binary (nullable = true)
 |    |    |-- sig_v: binary (nullable = true)
 |    |    |-- sig_r: binary (nullable = true)
 |    |    |-- sig_s: binary (nullable = true)
 |    |    |-- sendAddress: binary (nullable = true)
 |    |    |-- hash: binary (nullable = true)
 |-- uncleHeaders: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- parentHash: binary (nullable = true)
 |    |    |-- uncleHash: binary (nullable = true)
 |    |    |-- coinBase: binary (nullable = true)
 |    |    |-- stateRoot: binary (nullable = true)
 |    |    |-- txTrieRoot: binary (nullable = true)
 |    |    |-- receiptTrieRoot: binary (nullable = true)
 |    |    |-- logsBloom: binary (nullable = true)
 |    |    |-- difficulty: binary (nullable = true)
 |    |    |-- timestamp: long (nullable = false)
 |    |    |-- number: decimal(38,0) (nullable = true)
 |    |    |-- numberRaw: binary (nullable = true)
 |    |    |-- gasLimit: decimal(38,0) (nullable = true)
 |    |    |-- gasLimitRaw: binary (nullable = true)
 |    |    |-- gasUsed: decimal(38,0) (nullable = true)
 |    |    |-- gasUsedRaw: binary (nullable = true)
 |    |    |-- mixHash: binary (nullable = true)
 |    |    |-- extraData: binary (nullable = true)
 |    |    |-- nonce: binary (nullable = true)                                                                                                          

Format: org.zuinnote.spark.ethereum.block (enrich=false)

root
 |-- ethereumBlockHeader: struct (nullable = true)
 |    |-- parentHash: binary (nullable = true)
 |    |-- uncleHash: binary (nullable = true)
 |    |-- coinBase: binary (nullable = true)
 |    |-- stateRoot: binary (nullable = true)
 |    |-- txTrieRoot: binary (nullable = true)
 |    |-- receiptTrieRoot: binary (nullable = true)
 |    |-- logsBloom: binary (nullable = true)
 |    |-- difficulty: binary (nullable = true)
 |    |-- timestamp: long (nullable = false)
 |    |-- number: decimal(38,0) (nullable = true)
 |    |-- numberRaw: binary (nullable = true)
 |    |-- gasLimit: decimal(38,0) (nullable = true)
 |    |-- gasLimitRaw: binary (nullable = true)
 |    |-- gasUsed: decimal(38,0) (nullable = true)
 |    |-- gasUsedRaw: binary (nullable = true)
 |    |-- mixHash: binary (nullable = true)
 |    |-- extraData: binary (nullable = true)
 |    |-- nonce: binary (nullable = true)
 |-- ethereumTransactions: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- nonce: binary (nullable = true)
 |    |    |-- value: decimal(38,0) (nullable = true)
 |    |    |-- valueRaw: binary (nullable = true)
 |    |    |-- receiveAddress: binary (nullable = true)
 |    |    |-- gasPrice: decimal(38,0) (nullable = true)
 |    |    |-- gasPriceRaw: binary (nullable = true)
 |    |    |-- gasLimit: decimal(38,0) (nullable = true)
 |    |    |-- gasLimitRaw: binary (nullable = true)
 |    |    |-- data: binary (nullable = true)
 |    |    |-- sig_v: binary (nullable = true)
 |    |    |-- sig_r: binary (nullable = true)
 |    |    |-- sig_s: binary (nullable = true)
 |-- uncleHeaders: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- parentHash: binary (nullable = true)
 |    |    |-- uncleHash: binary (nullable = true)
 |    |    |-- coinBase: binary (nullable = true)
 |    |    |-- stateRoot: binary (nullable = true)
 |    |    |-- txTrieRoot: binary (nullable = true)
 |    |    |-- receiptTrieRoot: binary (nullable = true)
 |    |    |-- logsBloom: binary (nullable = true)
 |    |    |-- difficulty: binary (nullable = true)
 |    |    |-- timestamp: long (nullable = false)
 |    |    |-- number: decimal(38,0) (nullable = true)
 |    |    |-- numberRaw: binary (nullable = true)
 |    |    |-- gasLimit: decimal(38,0) (nullable = true)
 |    |    |-- gasLimitRaw: binary (nullable = true)
 |    |    |-- gasUsed: decimal(38,0) (nullable = true)
 |    |    |-- gasUsedRaw: binary (nullable = true)
 |    |    |-- mixHash: binary (nullable = true)
 |    |    |-- extraData: binary (nullable = true)
 |    |    |-- nonce: binary (nullable = true)