Is LevelDB 2 times faster than BadgerDB? (Update: No)

by Emile `iMil' Heitor - 2019-12-19

Update (2020/05/21) the method used in this post is totally sub-performant, and I finally found out about LevelDB’s and Badger’s batch methods, which make writes considerably faster, I’ll probably write another note about this.
And by the way, I found Badger to be much faster at writing batches than LevelDB.

Actual post

I’m working on a plugin for Goxplorer that will create a database of all Bitcoin addresses present in its blockchain.

That’s an exercise I already did using LevelDB, which is Bitcoin’s choice for some of its own data, and as the task took quite a while, I decided to give a shot to BadgerDB, which I cite is a fast key-value (KV) database written in pure Go.

Well, I must do something very wrong, because I get the following results:

BadgerDB

$ time ./goxplorer -t -b blk01845.dat -a -x -bc mkaddrdb
./goxplorer -t -b blk01845.dat -a -x -bc mkaddrdb  48.59s user 3.98s system 81% cpu 1:04.72 total

LevelDB

$ time ./goxplorer -t -b blk01845.dat -a -x -bc mkaddrdb
./goxplorer -t -b blk01845.dat -a -x -bc mkaddrdb  35.91s user 4.28s system 115% cpu 34.763 total

That’s embarrassing.

Maybe you’ll spot something terribly wrong in my code:

BadgerDB

func recAddrsToBdg(h []byte, addrs []string) {
	opts := badger.DefaultOptions("badgeraddr")
	opts.Logger = nil
	db, err := badger.Open(opts)
	fatalErr(err)
	defer db.Close()

	var blocks []byte

	err = db.Update(func(txn *badger.Txn) error {
		for _, a := range addrs {
			if len(a) == 0 {
				continue
			}
			item, err := txn.Get([]byte(a))
			// address not found, record it
			if err == badger.ErrKeyNotFound {
				err = txn.Set([]byte(a), h)
				fatalErr(err)
				continue
			}
			fatalErr(err)

			err = item.Value(func(val []byte) error {
				blocks = append([]byte{}, val...)
				return nil
			})
			fatalErr(err)

			// if block hash is not yet recorded, record it
			if !bytes.Contains(blocks, []byte(h)) {
				blocks = append(blocks, h...)
				err = txn.Set([]byte(a), blocks)
			}
		}
		return nil
	})
	fatalErr(err)
}

LevelDB

func recAddrsToLvl(h []byte, addrs []string) {
	db, err := leveldb.OpenFile("./addresses", nil)
	fatalErr(err)
	defer db.Close()

	var blocks []byte

	for _, a := range addrs {
		if len(a) == 0 {
			continue
		}
		blocks, err = db.Get([]byte(a), nil)
		// address not found, record it
		if err == leveldb.ErrNotFound {
			err = db.Put([]byte(a), h, nil)
			fatalErr(err)
			continue
		}
		fatalErr(err)

		// if block hash is not yet recorded, record it
		if !bytes.Contains(blocks, []byte(h)) {
			blocks = append(blocks, h...)
			err = db.Put([]byte(a), blocks, nil)
		}
	}
}

And yes, the number of keys is strictly the same:

BadgerDB

$ ../badger-cli/badger-cli list -d badgeraddr|tail -1
Matched keys:    482582

LevelDB

$ ../go-leveldbctl/leveldbctl --dbdir=addresses k|wc -l
482582

Or is just LevelDB 2 times faster than BadgerDB? ;)