Background

I come from C++/Python/NodeJS/Typescript development and want to switch fully to go to replace my tooling for command lines and web service using go. During my journey the web service started to grow and I tried to understand the mechanism of growth. Here are my notes.

The journey starts

When you start a go web server you could start as simple as a single main function.

package main

import (
    "fmt"
    "html"
    "log"
    "net/http"
)

func main() {

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "welcome to %q", html.EscapeString(r.URL.Path))
    })

    http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request){
        fmt.Fprintf(w, "pong")
    })

    log.Fatal(http.ListenAndServe(":8080", nil))

}

From a file system perspective your project will the look like this:

<project>/
  main.go
  ...

After adding database logic and a service layer your code will look roughly like this.

main.go
server.go
routes.go
handler.go
service.go
db.go

You could create a struct for each file or use one server struct for all. Encapsulation is not a big thing yet. By now you will also added a router such as chi with an own routes file.

// handler.go
package main

func (s *server) HandlePing(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintf(w, "pong")
}

Add command line and configurations

Now to add more features you will add proper command handling using cobra and config handling using viper. So adding a cmd/root.go and a config.go for loading and accessing the config files. By now you slowly move towards the 12 Factor principles.

cmd/root.go
config.go
db.go
handler.go
routes.go
main.go
server.go
service.go

Clean architecture

By now your application grows and you get demand for better encapsulation and abstraction. The clean architecture will come up in the discussions. To minimal fulfil the requirements just add a model.go which encapsulates your data entities.

cmd/root.go
config.go
db.go
handler.go
routes.go
main.go
model.go
server.go
service.go

Grow

When you code size growth you will start thinking about layers (e.g. model/repository/service/handler) and splitting all your different entities into own files.

cmd/root.go
model/user.go
model/team.go
model/product.go
db/user.go
db/team.go
db/product.go
...

or you take the domain approach

cmd/root.go
user/model.go
user/db.go
user/service.go
user/handler.go
user/routes.go

In the layer approach your dependencies are clear.

server -> handler/user -> service/user -> db/user -> model/user
server -> handler/team -> service/team (req. service/user)-> db/team -> model/team

In the domain approach it is not so clear. Especially if a functionality is required from a different module.

server -> user/handler -> user/service -> user/db -> team/model
server -> team/handler -> team/service (req. user/service) -> team/db -> team/model

In a domain approach your interdependencies are much harder to spot. So better keep a layer approach and scale out here.

The folder order in a domain approach would be domain -> (function per layer) The folder order in a layer approach would be layer -> (function per domain)

In a domain approach a change to one feature is easier to handle as you only work on one domain folder but what happens if you have not only a REST API with http handler but also a GRPC with GRPC services on top? Would you add a GRPC file to each domain folder?

Scale out

To scale out you create a layer approach inside applications. But then your domains should identify really applications not data entities. Think about little micro services you would create which are independent and could run on its own. Only linked by interfaces.

identity/model/user.go
identity/model/team.go
identity/db/user.go
identity/db/team.go
identity/service/register.go
identity/service/login.go
identity/handler/register.go
identity/app.go
product/model/product.go
...

So the folder order is more app -> layer -> (function per domain inside app scope)

Folder Structure Discussions

When a source code system growths it starts to be very difficult to find names which don't overlap. For example a model/user.go vs service/user.go. When having the document open in the editor it takes a second look to understand the difference.
I am a big fan of using longer dot separated word for file names (e.g. model/user.model.go and service/user.service.go) it makes it always clear what is the purpose of the file I am currently in. (se also NestJS.
Thanks to packing system of go a struct implementation can be spread about several files and file names don't really matter. Which allows you to have a user.service.go and maybe a team.service.go to share the same service struct.
Don't think about interfaces too much. Thanks to the implicit go interface you can always add interfaces when you need it, e.g. for the tests to create mocks or on the app level.
Try to avoid a folder structure which are deeper than two folders working on the files will get harder. Better to flatten out the structure.
Think about a folder as a package which shares similar dependencies and provides functionality related to the package name.

Summary

Scaling web applications is done best by rewriting the app logic. It is important to experience the growth in code and the needs for larger structures. Here we present just a simple idea how to structure go code to keep growing.

api-driven