A custom split function for bufio#Scanner

Custom implementation of a buffered scanner

I am currently working on a project that was originally a proof of concept i.e I am now to convert it to a real application. In it’s original form, it lacked tests, so I figured out that would be the first place I start from while refactoring and adding newer features.

While I was writing integration tests for mysql, I needed to be able to setup the database by creating appropriate schemas and all of that. Ideally, I would have used migrations that would run during app startup and tests initialization. But I didn’t have the luxury here 0, so I just moved the entire sql dump to testdata/init.sql.

The problem now was how to import the file into the database (during CI, as I obviously wouldn’t have access to the MySQL shell to run source file), I reached out for bufio#Scanner to implement some parsing but figured out it would not serve my usecase as it defaults to splitting content by lines. See here . I needed to be able to distinguish sql statements. A valid one for instance might span multiple lines and terminated by a semi-colon (;). I then decided to write a custom splitting function..

See this gist for a sample sql dump

Here is what I ended up with for the splitting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
func TestDB_KeyExists(t *testing.T) {

	db, _ := sql.Open("user:passwd@tcp(localhost:3306)/test?parseTime=true")

	f, err := os.Open("testdata/init.sql")
	require.NoError(t, err)

	buf := bufio.NewScanner(f)

	buf.Split(func(data []byte, atEOF bool) (int, []byte, error) {

		// Trim out unnecessary whitespaces
		trimSpaces := func(b []byte) []byte {
			return bytes.TrimSpace(b)
		}

		if len(data) == 0 {
			return 0, nil, nil
		}

		// SQL statements are delimited by ';'
		if i := bytes.IndexByte(data, ';'); i >= 0 {
			return i + 1, trimSpaces(data[0:i]), nil
		}

		if atEOF {
			return len(data), trimSpaces(data), nil
		}

		return 0, nil, nil
	})

	for buf.Scan() {
		db.Exec(buf.Text())
	}

	// Real test comes here
}

Hopefully this helps someone trying to implement something of this sort

Footnotes

[0] The major reason why I couldn’t use migrations was because the app uses an existing database..If I have made the dump the first migration, I risk having errors on running it against the production database as indexes that already exist would refuse to be created again and thus the app would refuse to start

comments powered by Disqus

My Newsletter

I send out an email every 2 weeks or there about often to inform you of articles I have written or cool stuff I'm working on and/or launching. Sounds like fun? go ahead and sign up

    We won't send you spam. Unsubscribe at any time.

    Powered By ConvertKit