Is LevelDB 2 times faster than BadgerDB?

I’m working on a plugin for Goxplorer that will create a database of all Bitcoin addresses present in its blockchain.

That’s an exercise I already did using LevelDB, which is Bitcoin’s choice for some of its own data, and as the task took quite a while, I decided to give a shot to BadgerDB, which I cite is a fast key-value (KV) database written in pure Go.

Well, I must do something very wrong, because I get the following results:

BadgerDB

1
2
$ time ./goxplorer -t -b blk01845.dat -a -x -bc mkaddrdb
./goxplorer -t -b blk01845.dat -a -x -bc mkaddrdb 48.59s user 3.98s system 81% cpu 1:04.72 total

LevelDB

1
2
$ time ./goxplorer -t -b blk01845.dat -a -x -bc mkaddrdb
./goxplorer -t -b blk01845.dat -a -x -bc mkaddrdb 35.91s user 4.28s system 115% cpu 34.763 total

That’s embarrassing.

Maybe you’ll spot something terribly wrong in my code:

BadgerDB

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
func recAddrsToBdg(h []byte, addrs []string) {
opts := badger.DefaultOptions("badgeraddr")
opts.Logger = nil
db, err := badger.Open(opts)
fatalErr(err)
defer db.Close()

var blocks []byte

err = db.Update(func(txn *badger.Txn) error {
for _, a := range addrs {
if len(a) == 0 {
continue
}
item, err := txn.Get([]byte(a))
// address not found, record it
if err == badger.ErrKeyNotFound {
err = txn.Set([]byte(a), h)
fatalErr(err)
continue
}
fatalErr(err)

err = item.Value(func(val []byte) error {
blocks = append([]byte{}, val...)
return nil
})
fatalErr(err)

// if block hash is not yet recorded, record it
if !bytes.Contains(blocks, []byte(h)) {
blocks = append(blocks, h...)
err = txn.Set([]byte(a), blocks)
}
}
return nil
})
fatalErr(err)
}

LevelDB

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
func recAddrsToLvl(h []byte, addrs []string) {
db, err := leveldb.OpenFile("./addresses", nil)
fatalErr(err)
defer db.Close()

var blocks []byte

for _, a := range addrs {
if len(a) == 0 {
continue
}
blocks, err = db.Get([]byte(a), nil)
// address not found, record it
if err == leveldb.ErrNotFound {
err = db.Put([]byte(a), h, nil)
fatalErr(err)
continue
}
fatalErr(err)

// if block hash is not yet recorded, record it
if !bytes.Contains(blocks, []byte(h)) {
blocks = append(blocks, h...)
err = db.Put([]byte(a), blocks, nil)
}
}
}

And yes, the number of keys is strictly the same:

BadgerDB

1
2
$ ../badger-cli/badger-cli list -d badgeraddr|tail -1
Matched keys: 482582

LevelDB

1
2
$ ../go-leveldbctl/leveldbctl --dbdir=addresses k|wc -l
482582

Or is just LevelDB 2 times faster than BadgerDB? ;)