Golang: Ensuring Data Integrity with Access Proxies

Dealing with an ever changing data can be tough. Especially when you’re trying to keep track of the original values at a given time. Let me explain:

Imagine a user is trying to submit a form which he filled with his desired inputs. Before finalizing the submit process and saving the data in a database, we want to either make sure about the validity of the inputted data or make certain modifications on it.

Let’s say we have 4 independent pieces of code (agents), 2 of which validates the data while the other 2 make modifications. Having that each agent is independent from the others, they are required to see the original input data without any modifications (think of Isolation in databases, one of the ACID properties. We don’t want any dirty reads, lost updates or similar problems. Serializability is the desired behavior in our case).

Whether or not we decide to run the agents concurrently, we have to make sure of the following:

Each agent acts on the original data
Since agents are independent, the order in which they modify or validate the data should not effect the final outcome. Meaning that we need to implement a way to capture the changes that each agent make to the original data and (probably) throw an error in case of any overlapping changes (since this would simply cause lost updates. Original → Change #1 → Change #2 (final) vs Original → Change #2 → Change #1 (final) hence non-identical outcomes)

In order to satisfy the first criteria, we have the obvious option to pass a copy of the original data to each agent, but is it scalable? Probably not. The memory overhead has a linear correlation with both the size of the input data and the number of agents, so it basically increases exponentially. So, we’re required to utilize pointers in order to prevent unnecessary allocations. We’ve spoken about pointers and their dangerous nature in a previous blog, so when dealing with them, we can’t be too cautious. Here comes the Access Proxy.

Disclaimer: I regularly make mistakes on a daily basis and I really learn from those who kindly correct my mistakes. If you noticed one, I’d be more than grateful if you would correct me.

Golang: Ensuring Data Integrity with Access Proxies

Golang: Ensuring Data Integrity with Access Proxies

Access Proxy

Let’s see what we need from an ideal access proxy. The first and probably the most obvious thing is that it should restrict direct access to its underlying data (hence the name, proxy). That’s easily achievable by a pair of getter/setter methods. Imagine the structure below, for which we’re trying to write an access proxy:

type Data struct {
  FirstField string
  SecondField int
}

From the stand point of the agents, the access proxy can be used like this:

type AccessProxy sturct {...}

type Agent interface {
  run(*AccessProxy) error
  name() string
}

func runAgents(data *Data, agents []Agent) error {
  ap := NewAccessProxy(data)
  for _, agent := range agents {
    err := agent.run(ap)
    if err != nil {
      return fmt.Errorf("failed to run agent `%s`", agent.name())
    }
  }

  // apply changes made by individual agents
  err := ap.ApplyChanges()
  handle(err)

  return nil
}

The code snippet above instantiates a new AccessProxy on the underlying data and runs each agent with the same access proxy.

The bare-bones access proxy might look something like below:

type AccessProxy struct {
  data *Data
}

func (p *AccessProxy) GetFirstField() string {
  return p.data.FirstField
}

func (p *AccessProxy) SetFirstField(v string) {
  p.data.FirstField = v
}

func (p *AccessProxy) GetSecondField() int {
  return p.data.SecondField
}

func (p *AccessProxy) SetSecondField(v int) {
  p.data.SecondField = v
}

But wait, we’re essentially letting any given agent directly change the underlying data! This makes the ordering of the agents matter (something that we REALLY don’t want) and if run concurrently, race conditions can happen. Instead of actually making the change, we can send a change signal:

type AccessProxy struct {
  changeSigs []changeSig
  data *Data
}

func NewAccessProxy(data *Data) *AccessProxy {
  return &AccessProxy{
            data: data,
            changedFields: make(map[string]struct{}),
          }
}

type changeSig interface {
  fieldName() string
}

type firstFieldChangeSig struct {
  val string
}
func (s FirstFieldChangeSig) fieldName() string {return "FirstField"}

func (p *AccessProxy) SetFieldField(v string) {
  p.changeSigs = append(p.changeSigs, firstFieldChangeSig{val: v})
}

Let’s review what just happened here. Instead of applying the change right as theSetX method is called, we fire a change signal and store it somewhere. This way we can apply the whole changes after all the agents finished their job. This way we’ve guaranteed that each agent have access to the original data, and at the same time, provided ourselves with a way to examine duplicate and overlapping changes made by different agents. This aggregation can be achieved by the ApplyChanges method:

type AccessProxy struct {
  changeSigs []changeSig
  data *Data
  changedFields map[string]struct{}
}

func (p *AccessProxy) ApplyChanges() error {
  for _, s := range p.changeSigs {
  
    // check for duplicate changes
    if _, alreadyChanged := p.changedFields[s.fieldName()]; alreadyChanged {
      return fmt.Errorf("duplicate changes detected for `%s`", s.fieldName())
    }

    p.changedFields[s.fieldName()] = struct{}{}
  
    switch sig := s.(type) {
    case firstFieldChangeSig:
      p.data.FirstField = sig.val
    case secondFieldChangeSig:
      p.data.SecondField = sig.val
    default:
      // should not happen
      return fmt.Errorf("unknown signal: %+v", sig)
    }
  }

  return nil
}

It seems like we’ve built a pretty standard access proxy. But are we done? What happens if we detect duplicate changes? Well, we stop the process and return an error (which is good) but it would be too late as we’ve already applied a bunch of changes on the underlying data. If we try to do anything else with the original data afterwards, we’ll have to deal with a half changed and inconsistent data. In other words, we need a rollback mechanism. In order to rollback, we can simply save the original data somewhere else in the proxy:

type AccessProxy struct {
  original Data
  data *Data
  ...
}

func NewAccessProxy(data *Data) *AccessProxy {
  return &AccessProxy{
           data: data,
           original: *data,
           changedFields: make(map[string]struct{}),
         }
}

func (p *AccessProxy) ApplyChanges() error {
  ...

  if somethingGoesWrong() {
    p.rollback()
    return errors.New("something went wrong")
  }
  
  ...
}

func (p *AccessProxy) rollback() {
  *p.data = p.original
}

Let’s try and see if it works:

func main() {
  d := &Data{
          FirstField: "first field",
          SecondField : 1,
        }
  
  ap := NewAccessProxy(d)
  
  ap.SetFirstField("changed first field")
  ap.SetSecondField(2)

  err := ap.ApplyChanges()
  if err != nil {
    fmt.Printf("data: %+v\n", data)
    log.Fatal(err)
  }

  fmt.Printf("data: %+v\n", data) // expect "changed first field" and 2
}

Needless to say that if we try to change one of the fields for a second time, ApplyChanges will throw and error. Also, the underlying data (d) will be rolledback to it’s original form.

But wait! Does it really do what it’s intended to? Does it really prevent the user to change fields on demand without any protective layers? Let’s try a curious situation:

type Data struct {
  ...
  ThirdField *string // notice this new field
}

func (p *AccessProxy) GetThirdField() *string {
  return p.data.ThirdField
}

If you look closely, it becomes apparent that the pointer to the underlying data is being returned in this example, which can be dereferenced and changed on demand (but ironically unwanted):

func main() {
  d := &Data{
          ThirdFild: toPoiter("third field"),
        }
  
  ap := NewAccessProxy(d)
  
  f := ap.GetThirdField()
  *f = "BAD CHANGE"
  
  fmt.Println("third field:", *d.ThirdField) // third field: BAD CHANGE 
}

We should consider a slight but rather necessary change in our implementation in order to prevent these kinds of unwanted changes:

type Data struct {
  ...
  ThirdField *string
}

func (p *AccessProxy) GetThirdField() *string {
  copy := *p.data.ThirdField
  return &copy
}

Bonus tip: Consider utilizing mutexs and/or channels to ensure concurrency safety (notice the changeSigs []changeSig)

Are we done? I don’t think so. Do we really need to write every single signal, getter and setter by hand? Probably not!

In the next part of this blog series, we will discover the world of auto-generated code with Go using templates, as well as exploring convenient ways to extract the necessary metadata from a given struct, which can be later used in par with our templates in order to auto-generate working Go programs.

If you enjoyed this blog post, consider a clap and sharing it with the ones you care about. You can also find me on LinkedIn, Twitter and my Website. You can also find the code snippets in my Access Proxy Demo GitHub Repo.