Re: Making thread-join more consistent wrt 'signal' vs 'thread-signal'

[Eli Zaretskii]

> Date: Sat, 16 Aug 2025 18:11:03 +0300
> Cc: emacs-devel@gnu.org, michael.albinus@gmx.de
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 15/08/2025 09:47, Eli Zaretskii wrote:
> 
> > Why do you have to have these requirements fulfilled?  Only because of
> > convenience, or some other reasons?
> 
> These seem like basics for this operation's semantics. They are such in 
> other programming languages that I've seen, and it doesn't seem feasible 
> to write reliable programs using it. You said it yourself not too long ago:
> 
>    > error handling in non-main threads is generally rudimentary and
> barely useful.  For example, if a thread signals an error, it simply
> silently exits and leaves the error form in a variable that can be
> accessed via thread-last-error.  AFAIR, it is not trivial to improve
> the thread error handling significantly, but patches are welcome, of
> course.
> 
> (bug#65837)
> 
> Or here is another user request (from 2019) which would be solved by a 
> reliable thread-join: bug#37480.
> 
> I mentioned it before too, in 2021: 
> https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg02058.html
> 
> The response was "patches welcome", so this is a patch.

Existing requests for features are not a justification I was looking
for.  I was looking for some grave consequences of what we have now.
I don't yet see anything like that.

As for "patches welcome", I hope you agree that it should not be taken
as "any patch will be automatically accepted".

Let me just state the obvious, if you haven't figured that out
already: we don't have on board anyone who takes care of this part of
Emacs.  I've been fixing thread-related problems here and there since
this stuff has landed on master, because no one else would.  What we
have now might be not ideal, but the basic features work, and I'm not
aware of any glaring bugs (of which there were several in the
beginning).  So the only way I'll agree to changes that modify the
design and expand this further (i.e. not changes that fix clear bugs)
is if someone steps forward to be the responsible maintainer of this
part of Emacs, and will thereafter keep the promise by picking up any
and all bug reports and other discussions related to threads, and
working on them actively and efficiently.  I cannot be that person, so
someone else will have to volunteer.  Maybe you.

Until that happens, don't expect too much sympathy from me to any
changes in how this was originally designed.  Sorry to be blunt, but
it sounds like nothing else gets my point across, so much so that I'm
beginning to look like some stubborn ass who is an obstacle to
progress.  At least now I have my reasons spelled out: if we want
development and enhancements in this area, let the interested folks
step up to the job.

> The most basic example is:
> 
>    * create several threads
>    * make http requests with each (or process calls)
>    * call thread-join on each of them in the list
> 
> This has to error out if any of the threads has failed, no matter at 
> which point in a thread's lifecycle this had been done, otherwise the 
> general pattern does not work.

I disagree that it "has to error out, no matter what".  Emacs Lisp
threads are not just threads in some GP programming language.  They
have restrictions, which are basic in their design.  And if one of
those restrictions says that one must do the above in such-and-such
manner and no other, for it to work, I see no catastrophe.  People who
come from other languages and expect to get the same semantics with
Emacs Lisp threads have wrong expectations, and should adapt.

But yes, it would be nice to have the same semantics as in other
environments, if we could make it work, what with all the restrictions
we must observe.  Cf. what I say above about active development in
this area.

> >> I think the key issue is predictability of computation in the face of
> >> random thread scheduling.
> > 
> > I'm afraid you expect too much of what we have.  There are still some
> > unresolved issues there and some windows of race conditions.  See
> > bug#79201 as one recent example.
> 
> It's an example of another bug, which is now marked as fixed.

Did I say something different?  Once again, clear bugs will get
attention and their fixes will be installed, that's a no-brainer.  My
point is that, as long as problems of this kind exist, your
expectations of "predictability in the face of random thread
scheduling" are very far from what we have now.  Even reasoning about
threads in Emacs is not simple, and starkly different from what we
have in other languages.  For example, in this excerpt from our test
suite:

      (let ((threads ())
            (cat (executable-find "cat")))
        (skip-unless cat)
        (dotimes (i 10)
          (let* ((name (format "test %d" i))
                 (process (make-process :name name
                                        :command (list cat)
                                        :coding 'no-conversion
                                        :noquery t
                                        :connection-type 'pipe)))
            (push process processes)
            (set-process-thread process nil)
            (push (make-thread
                   (lambda ()
                     (while (accept-process-output process)))
                   name)
                  threads)))
        (mapc #'process-send-eof processes)
        (cl-loop for process in processes
                 and thread in threads
                 do
                 (should-not (thread-join thread))
                 (should-not (thread-last-error))
                 (should (eq (process-status process) 'exit))
                 (should (eql (process-exit-status process) 0)))))))

where in this code is the point where one of the non-main threads can
start running its Lisp function?  Bonus points for discussing the
implications of this on use of threads in Emacs as opposed to other
languages.

> > From where I stand, what you are
> > asking for is nowhere near the important problems we still have in
> > threads machinery.  So I would prefer not to make significant changes
> > for such problems for now.
> 
> The proposed patch is about 8 lines long. Is it really a significant 
> change or a fix?

Yes, it's significant from where I stand.  It modifies the design of
how errors were supposed to be reported by thread-join.

> None of the existing tests had to be changed, only new ones added.

Our tests wrt threads are really minimal, so I'm not surprised that
they are insufficient to pick up effects of such changes.  If
anything, that only demonstrates the low coverage of the test suite in
this area.

> Also note that there are zero (!) uses of thread-join in-tree. My 
> thinking is that its current behavior makes it less than useful.

The same is true for make-thread (and any other thread-related
primitive).  What does that tell us?

> To turn the table on your question, do *you* have a realistic example 
> where the current behavior would be preferred, rather than the one with 
> the patch?

I don't need to provide such an example, because, all else being
equal, the current behavior gets precedence, just by virtue of being
there for all this time.

> >>> How can this work reliably?  If the thread whose error is being
> >>> recorded here dies or exits by the time thread-join finishes waiting,
> >>> the thread_state of the thread with the recorded error is unchained
> >>> from the list of threads, and will be collected by GC at some point.
> >>
> >> As long as some other thread keeps a reference to this thread (to be
> >> able to call 'thread-join'), don't we keep the structure around?
> > 
> > Not after thread_join_callback returns, not reliably, I think?
> 
> Could you elaborate? What I see is looking up tstate in another thread 
> and switching to it. It doesn't have to follow that the reference to the 
> current one is lost.

You basically rely on GC to find the unchained thread_state object
somewhere on the C stack.  I don't see how this would have to happen
with 100% reliability.