Settings

Theme

Go nulls and SQL

ente.io

67 points by mnvrth 4 years ago · 76 comments

Reader

Mawr 4 years ago

Great, except:

1. The type is merely a weak hint that you should check .Valid before you use the value, there's no enforcement: https://go.dev/play/p/nS8RxGujMBk

It's much better when the struct members are private and the only way to access the value is through a method that returns (value, bool):

  if value, ok := optional.Get(); ok {
    // value is valid
  } else {
    // value is invalid
  }
This is a strong hint that you should check the bool before using the value. It's also a common go pattern - checking for existence of a key in a map is done this way.

2. We have generics now. Why use type-and-sql-specific wrappers when you could use a generic Option? Example implementation: https://gist.github.com/MawrBF2/0a60da26f66b82ee87b98b03336e....

  • psadauskas 4 years ago

    Your comment is helpful, so this sarcasm isn't directed at you. However, this looks like an extremely cumbersome way to wish your language had the Result monad.

    • eyelidlessness 4 years ago

      I upvoted both your comment and GP. Here, hopefully this helps your comment feel more productive: this is a Result type, albeit yes not a Monad. It’s (AFAIK) the idiomatic way the type is expressed in Go, so maybe cumbersome to implement but trivial to adopt.

      It’s also something I realized was lost in the JS progression towards Promise APIs and eventually async/await. Promises have ergonomic benefits over callback hell, but damn if the Node convention of cb: (error, result) => { /* … */ } wasn’t a Result type sitting there begging to be embraced. Again, not a Monad, but it’s a shame the good API design was thrown out with the inconvenient API design.

    • svnpenn 4 years ago

      > extremely cumbersome

      I'd say the opposite. I'd say it beats the hell out of Rust syntax, as Rust would force you to have staircase code here. No thank you.

      • saurik 4 years ago

        FWIW, that's because Rust also doesn't have a Result monad, it merely has a Result type: in Haskell, you would provide the actual operations of the Monad trait for your Result type--the way in which values are converted into and used from within the type: two functions called return and (>>=)--which would allow you to delegate the execution of that abstracted boilerplate to the language's do-notation.

tedunangst 4 years ago

> Pointers are easy first, but then you realize that you have to put nilness checks everywhere.

Well, yeah, if something may or may not be there, you have to check before you use it. How else do people want it to work?

  • solar-ice 4 years ago

    In some languages, the concept of "may or may not be there" is cleanly separated from "exists in a separate block of memory". There's plenty of pointers which will never be nil - checking that they're nil at the entry and exit points of every function is line noise.

    • throwaway894345 4 years ago

      I don't think the parent was advocating for checking for nil before/after each function, but rather noting that this option pattern doesn't buy you any more type safety in Go because there's nothing enforcing you to check appropriately any more than there is for a pointer.

      The salient rebuttal is that this pattern is a strong hint to check, whereas a pointer is ambiguous (it's unclear whether or not a bare pointer may ever be nil, but you would only use this pattern to be explicit that a check is needed). This pattern can also support value types so you don't have to worry as much about unnecessary allocations.

    • IshKebab 4 years ago

      Is "some languages" just Rust?

      • solar-ice 4 years ago

        Also Swift, and various functional languages too, although "exists in a separate block of memory" is basically the default there.

  • xdfgh1112 4 years ago

    Sane languages have pointers which either can't be null or require you to check explicitly. Rather than leaving it to explode at runtime.

    • tedunangst 4 years ago

      Tell me more about these unexplodable languages.

      • dtech 4 years ago

        If this question is in good faith, for example Kotlin differentiates between the nullable T? and the non-nullable T. An if x != null check automatically casts T? to T. You can't call a method on T?, preventing null pointer exceptions. C# and Typescript have similar language support. Other languages push users away from the problem by favoring monadic Optional/Maybe solutions.

        • tedunangst 4 years ago

          There are some differences here, but you can't for instance use a string pointer with string functions in go either. IMO The distinction between pointer and optional is not as vast as people make it out to be.

          (But thanks, I overlooked T? conversions inside if.)

          • ricardobeat 4 years ago

            No you can't, but Go will happily de-reference the pointer without a null check, and your "string" will blow up in production: https://go.dev/play/p/7EVa7q6VgIM

          • akavi 4 years ago

            The distinction between pointer and optional is not vast. The distinction between pointer and not optional is.

            The whole idea of optional is to make not optional possible.

          • morelisp 4 years ago

            You can call a value method on a nil pointer and it panics, though, which I think was a mistake.

          • nemothekid 4 years ago

            >The distinction between pointer and optional is not as vast as people make it out to be.

            The billion dollar mistake is overblown?

        • adastra22 4 years ago

          T? is just syntactic sugar for Optional/Maybe, no?

          • shirogane86x 4 years ago

            I don't think that's quite true. It's sort of the same in how it's used, but the main difference is that T? (or T | null) is a union, and Optional/Maybe/Option is a sum type. Out of all the implementations I've seen, T? can't be nested - you can't have T??, whereas it's quite possible to have a Option<Option<T>> (and it's sometimes useful).

          • MrJohz 4 years ago

            Roughly, yes. But that's the point: an Optional type along with static typing preventing misuse can prevent these sorts of errors entirely.

          • dtech 4 years ago

            It's semantically similar but not syntactic sugar, since it is not translated to Optional or wrapper types, so it also doesn't incur the overhead of Optional.

            I actually like the language support more, as it flows a lot better than the (flat)map chain you get in monadic style. Similar to async/await v.s. Future.

      • stouset 4 years ago

        Type systems can enforce that types always contain a value of that type and cannot be null. There is another type that is either null or a value of the enclosed type.

        At any boundary where something might be null, you do the null check. If it's null, you do whatever logic is necessary right there and only right there. That might be to skip the computation, to use a default value, to get the value from somewhere else, to return an error, or whatever. If it's not null, you use the internal value and from then on every user can operate on a guarantee that it's not null.

      • saghm 4 years ago

        >> Sane languages have pointers which either can't be null or require you to check explicitly. Rather than leaving it to explode at runtime.

        > Tell me more about these unexplodable languages.

        I don't think anyone else mentioned an entire language being unexplodable, just a pointer type.

        Later on the thread you talk about `Option` in Rust, which is not what I think most people would consider a pointer type, but even if it is, it's certainly not the only one. I think GP was talking about the basic reference type (i.e. `&T`), which will not ever be null in safe code, so dereferencing it will not explode.

      • adastra22 4 years ago

        Rust being the popular example. But even C++ has this with std::optional, for example.

        • tedunangst 4 years ago

          So rust doesn't explode if I unwrap None?

          • stouset 4 years ago

            Only if you want it to.

                #[derive(Default)]
                struct Foo {
                    …
                }
            
                struct Bar {
                    …
                }
            
                fn do_something(foo: Foo) -> Bar {
                    # do something with Foo and return a Bar
                    …
                }
            
                fn main() {
                    let opt_foo: Option<Foo> = None;
            
                    // panics
                    let foo: Foo = opt_foo.unwrap();
            
                    // panics, but with an error message of your choosing
                    let foo: Foo = opt_foo.expect("foo was supposed to be there");
            
                    // converts the Option (Some or None) to a Result (Ok or Err,
                    // where Err can contain an error type or message and then passed
                    // around or returned or whatever)
                    let foo: Result<Foo> = opt_foo.ok_or("foo was supposed to be there");
            
                    // replaces it with a value of your choosing if it's not there
                    let foo: Foo = opt_foo.unwrap_or(Foo { … });
            
                    // replaces it with the default value the type defines
                    let foo: Foo = opt_foo.unwrap_or_default();
            
                    // keeps it as `None`, or if it's actually something then
                    // replaces the internal contents with the result of the
                    // function call
                    let bar: Option<Bar> = opt_foo.map(do_something);
            
                    // does arbitrary logic in the match arm if foo is there
                    match opt_foo {
                        Some(foo) => { do something with foo },
                        None      => { do something else },
                    }
                }
            
            There are a few dozen other things you can do with an Option that handle the rarer use-cases, but the above are like 95%+ of what you want.
          • Gwypaas 4 years ago

            You're explicitly choosing to panic if it is None. Would never pass a code review unless you can in very clear terms tell why it will never now or in the future be none.

            The point is that you can't use the inner value without choosing a course of action if it happens to be none.

          • avgcorrection 4 years ago

            The comment that you respondend to:

            > > Sane languages have pointers which either can't be null or require you to check explicitly.

            You cannot try to dereference a nullable pointer in safe Rust. It has got nothing to do with Optional (Some/None).

            • tedunangst 4 years ago

              That would make a rust pointer rather less useful for representing sql null, no?

              • avgcorrection 4 years ago

                What?

                The comment that you -- with aloof disbelief -- replied to only talked about pointers. I don't know what point you think you are proving with these pithy replies. It sure does go above my head. :)

                • tedunangst 4 years ago

                  Well, this is a thread about storing sql null. I'm trying to suss out how sane languages represent sql null in a way that can't go wrong. So far the answers mostly seem to be use a type that can't represent sql null or use a type that explodes when you use it wrong. With a side of real programmers don't let bugs pass code review.

                  • stouset 4 years ago

                    With all due respect, it appears as if you have entered this conversation with this belief and have used that perspective as a filter when other commenters have tried to correct that perspective.

                    It may be surprising to learn that languages without implicitly-nullable types are real, but you should understand that none of these people would be trying to tell you about this if these languages simply exploded any time you used them the wrong way.

                  • avgcorrection 4 years ago

                    > I'm trying to [sass] out how sane languages represent sql null in a way that can't go wrong.

                    There is a difference between a type Pointer where `null` is an instance of that type and a type Box which only has valid (can be dereferenced) values. If the language has e.g. algebraic data types and pattern matching then Option(al)<Box> can be safely matched on and Box can be used in the branch (the pattern match arm) where Some(Box). Meanwhile in a language with Pointer and no flow analysis any dereference of Pointer could lead to some kind of "panic".

                    Say "unwrap()" all you want but the two approaches are clearly very different.

                  • winstonewert 4 years ago

                    Where did anyone anywhere in this thread claim that you could "represent sql null in a way that can't go wrong?"

                    Yes, Rust options explode if you unwrap a None. But that's explicitly what `unwrap()` does. It a big improvement over any random access potentially implicitly exploding.

                  • anonymoushn 4 years ago

                    It seems like sql types that include null and non-null values are a union of at least two different types. In general, if you write code to handle a tagged union that has "panic!" in one of the prongs, you'll get panics. Don't do that?

              • imron 4 years ago

                Correct, which is why you'd use Option<T> instead.

          • Dylan16807 4 years ago

            The quote is talking about plain pointers being dangerous.

            And in turn, the claim above was that some languages have plain pointers that aren't dangerous.

            Yes, you can put a pointer into an Option that can fail, but that's a different issue. The point is that "can fail" is not built directly into the pointer type, and you can use pointers that don't have that risk.

            And to be clear, that claim is not directly addressing the issue of SQL null. It's a baseline discussion of what pointers actually imply by nature of being pointers. We can then better build upon that knowledge after we separate "refers to a data location" and "might not have contents" into separate attributes.

      • anonymoushn 4 years ago

        Zig encodes optionality separately from pointer-ness, like Rust. You don't have to ask about unwrapping None in cases where None is not a member of &T.

      • worik 4 years ago

        You cannot use a null pointer in Rust.

        I am a Rust novice but I have not found a way to get to uninitialised memory in safe Rust. Yet.

        • imron 4 years ago

          > You cannot use a null pointer in Rust.

          Yes you can: https://doc.rust-lang.org/std/ptr/fn.null.html

          You just won't use it unless you're doing FFI or similar things.

          • worik 4 years ago

            FFI is unsafe, by definition.

            • imron 4 years ago

              You can still use null pointers in safe rust (see the link I posted which is 100% safe rust), you just can’t deference then.

              • worik 4 years ago

                Ok.

                I meant "using null pointer" means dereference it. But careless language....

        • tedunangst 4 years ago

          How do you create a pointer to uninitialized memory in go?

          • TheDong 4 years ago

            All it takes is a specially formatted comment

                /*
                int g() {
                 volatile int x;
                 return x;
                }
                */
                import "C"
            
                import "fmt"
            
                func main() {
                 fmt.Printf("%d\n", C.g())
                }
            
            
            But I know you'll say that 'import "C"' or 'import "unsafe"' is the same thing as using an unsafe block in rust or such, and really shouldn't count against go.

            Which is fair and true, but you're chasing down a pointless detail. The point isn't that go is memory unsafe. It's not. The point is that Go's type-system is not powerful enough to express various types of type-safety, and as such it's an error-prone language where you can expect null pointer exceptions frequently.

          • Gwypaas 4 years ago

            You can't. But a runtime panic also very easily causes an outage. At least it's not as subtle.

            Still. Rust is infinitely better in this regard. Go feels like a crude hammer, especially regarding the default interaction between deserialization and missing struct members.

          • stouset 4 years ago

            You can't. But you can absolutely have a `nil` that will unavoiad runtime panics if you don't check it. And further, you only can't do this because go defines by fiat that the zero-value of a type must be legal. Of course, this is wildly inconvenient and annoying for many types, including those that come bundled in the standard library.

            There are other, very convincingly better options to this that many in this thread have been trying to teach you about in the expectation that your responses have been in good faith.

      • giraffe_lady 4 years ago

        Maybe Some other time.

  • anoctopus 4 years ago

    If something may or may not be there, that should be a clear part of its type that the compiler checks you handled (even if it's just you deciding to panic, because sometimes that's the right thing to do). Then there can be a type for just the thing, no hidden possibility of it not being there that you have to check for, and almost all code can be written for the type that is guaranteed to be there, with only a little code dealing with only possibly-there values.

davidkuennen 4 years ago

Using go and postgres for my App's backend.

After using NULLs this way at first, I noticed it's generally much much easier for me to just avoid nullable SQL columns wherever possible (it was always possible so far). Most of the time there is a much easier was to say a value is empty. For strings '' for example.

This seriously made everything so much easier. Not necessarily anything to do with go tho.

  • sa46 4 years ago

    That's the approach I've taken as well. We use coalesces to ensure queries to never return null and handle the zero value or a sentinel value specially. Since our API layer is protobufs which don't have nil, the zero value is correct the majority of the time.

  • SoftTalker 4 years ago

    I think this is generally good, unless your database treats NULL and '' as the same thing.... e.g. Oracle.

  • layer8 4 years ago

    > This seriously made everything so much easier.

    Except inner joins and the like, I guess.

  • turboponyy 4 years ago

    You should disallow nulls as much as possible, but "" is just as valid of a string value as "John Smith"; if the field is actually semantically nullable, reflect that in the type - don't use arbitrary "blank" values to denote nulls.

    • layer8 4 years ago

      A string being empty oftentimes isn’t semantically different from the value being absent. I’d argue that, this being the case, the type should also be able to reflect whether the string can be empty or not (or all-whitespace or not, etc.).

      • turboponyy 4 years ago

        I completely agree, but conventional database systems don't support dependent typing unfortunately. Given the constraints, the best you can do in this situation whilst remaining semantically consistent in the type signatures is to leave the field nullable and enforce value checks (e.g. can't be blank).

        • layer8 4 years ago

          Database systems support constraint checks. Given that SQL isn’t statically typed, that’s almost the best you can do. The only lack is that you usually can’t bind constraints to a user-defined type name, so you have to repeat the constraint definition for each relevant column/table. But on the programming language side, dependent types (or something equivalent) would be appropriate, and I expect will become common practice at some point in the future.

          • turboponyy 4 years ago

            I think we may have been talking past each other - I did mean SQL's CHECK() et al with "value checks". I agree with everything you've said.

            • layer8 4 years ago

              Right, I missed “remaining semantically consistent in the type signatures”. Assuming you mean correspondence between the DDL schema definition and the PL type signature, I see no need to do so, as long as both can be generated from the same definition, or one from the other. That is, the scenario were dependent types are used on the PL side and constraint checks on the database side seems perfectly fine to me (given the limitations of SQL). I’m assuming that you meant value checks on the PL side if you didn’t mean checks on the DB side.

              • turboponyy 4 years ago

                By "type definition" I did actually mean the database schema. I don't particularly care what happens at the boundaries, and I think translating a database schema into suitable types at the programming language level makes sense (e.g. dependent types). I was just describing how it would make most sense in my mind to design the database schema in isolation with other concerns in order to remain most semantically correct.

      • JyB 4 years ago

        This is almost always true but somehow often overlooked. There's almost always no reason to rely on "null" to mean "empty". A carefully chosen default value often does the job.

hans_castorp 4 years ago

> by declaring our SQL columns as NOT NULL when possible.

I hear this advice quite often, but that doesn't relief you of handling NULL values. E.g. a NOT NULL column can be NULL in a result of an outer join.

eknkc 4 years ago

So, to avoid checking for null, you'll check for `NullString.Valid` now? The string pointer is a part of the language. You can pass it around, expose as part of a library etc. And it conveys the intent perfectly.

I have no idea what is the issue here?

  • psanford 4 years ago

    The nice thing about the Null* type is they can reduce the number of allocations done on the heap and thus also reduce total GC your program needs to do.

    • legorobot 4 years ago

      I haven't thought about it that way! I've not hit the GC as a performance wall when it comes to accessing nullable DB values yet. Thanks for the insight.

      • throwaway894345 4 years ago

        It's generally useful, not just in SQL. Any time you have a type that could be multiple things, you have to choose between a reference-based implementation (e.g., interfaces) or a tagged struct (a struct with a flag field that tells you what its runtime type). The tagged struct version is guaranteed not to allocate, while the interface version very likely will. Most of the time it will only matter if you're in a tight loop.

  • legorobot 4 years ago

    I think the idea is `sql.NullString` can be used to have a SQL NULL still be an empty string in Go (just avoid checking the `Valid` field, or cast to string -- no check necessary).

    It seems like the intent of a string pointer vs. `sql.NullString` is their goal with this type anyways[1]?

    In practice I've used a string pointer when I need a nullable value, or enforce `NOT NULL` in the DB if we can't/don't want to handle null values. Use what works for you, and keep the DB and code's expectations in-sync.

    [1]: https://groups.google.com/g/golang-nuts/c/vOTFu2SMNeA/m/GB5v...

  • catlifeonmars 4 years ago

    Sometimes you want a non-nullable string pointer. In Go, nullability and dereferencing are coupled under the same language construct (pointers).

jackielii 4 years ago

I have an alternative solution to work with an existing struct that you can't change, e.g. protobuf generated code: https://jackieli.dev/posts/pointers-in-go-used-in-sql-scanne...

  type nullString struct {
    s *string
  }
  
  func (ts nullString) Scan(value interface{}) error {
    if value == nil {
        *ts.s = "" // nil to empty
        return nil
    }
    switch t := value.(type) {
    case string:
        *ts.s = t
    default:
        return fmt.Errorf("expect string in sql scan, got: %T", value)
    }
    return nil
  }
  
  func (n *nullString) Value() (driver.Value, error) {
    if n.s == nil {
        return "", nil
    }
    return *n.s, nil
  }
Then use it:

  var node struct { Name string }
  db.QueryRow("select name from node where id=?", id).Scan(nullString(&node.Name))
brushyamoeba 4 years ago

I'm surprised this doesn't discuss JSON marshalling

cratermoon 4 years ago

Yea I ran into this the first time I wrote much Go code to deal with SQL databases. After being initially annoyed I realized how much more correct it is for the SQL interface code to check .Valid and write proper, meaningful special case/null object handling for the domain types.

AtNightWeCode 4 years ago

Maybe I passed out among the nonsense but most langs provide a dbnull constant to check against, no?

  • jrockway 4 years ago

    You could do that in Go, but it's not what database/sql's API supports. You read columns out of each row with "row.Scan(&col1, &col2, ...)". The types of col1 and col2 are declared at compile time, and they don't have to be able to represent the concept of null. So there would be no way to store the state that represents that something was null.

    You could of course have an API that just returns a slice of "any", and conditionally check whether a value is of type "string" or "mylibrary.NullValue" after the fact. This isn't clearly better to me than the Scan API. You are going to have to eventually cast "any" to a real type in order to use it; with Scan the library does that for you.

    Your own types can implement sql.Scanner to control exactly how you want to handle something. (Indeed, your "Scan" method receives something with type "any", and you need to check what the type is and convert it to your internal representation.)

    Also wanted to throw this out here; you don't have to be satisfied with lossy versions of your database's built in types. Libraries like https://pkg.go.dev/github.com/jackc/pgtype@v1.11.0 will give you a 1:1 mapping to Postgres's types. (I'm sure other database systems have similar libraries.)

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection