Root-cause ores.wt.service's intermittent glibc pthread crash
Table of Contents
This page is a capture in the inbox bucket of the product backlog — a pre-sprint idea, not yet pulled into a sprint as a story.
What
ores.wt.service's Wt httpd crashed intermittently on startup with a
fatal glibc error:
Fatal glibc error: tpp.c:83 (__pthread_tpp_change_priority): assertion failed: new_prio == -1 || (new_prio >= fifo_min_prio && new_prio <= fifo_max_prio)
This fires deep inside glibc's priority-protected mutex code
(PTHREAD_PRIO_PROTECT=/=INHERIT) — nothing in ORE Studio's own code
sets thread priorities, so this is coming from Wt's compiled httpd or a
Boost internal it depends on. The bright_faraday environment where
this was diagnosed has ulimit -r (real-time priority) at 0, both
soft and hard limit, which is exactly the condition that trips this
specific assertion — but it's unconfirmed whether that ulimit is
specific to that sandbox or also true on regular dev machines, since
the reporting user says the service "seems to have this problem
frequently" more broadly.
The controller's existing restart/backoff (retry_strategy) already
self-heals eventually (it did recover after ~8 attempts in the
diagnosed case), so this isn't a hard outage — just a noisy,
minutes-long crash-loop on every environment boot where it's hit.
As a stopgap (PR #1592), ores.wt.service is disabled by default in
controller_service_definitions_populate.sql (enabled = 0), so fresh
environments no longer hit this crash loop at all. Re-enabling is a
one-line SQL update, documented inline in that file.
Why
Root-causing needs an actual debugger session (attach gdb, or enable
core dumps and inspect a captured crash) against Wt/Boost's compiled
internals to find the exact call site — that's not something safely
doable from static analysis alone, and wasn't in scope for the
hotfix. Until it's done, ores.wt.service (the Wt-framework web UI)
stays off by default, which is an acceptable stopgap but not a
permanent state — someone should either fix the actual crash or
determine it's an environment-only quirk (e.g. sandbox-specific
RLIMIT_RTPRIO) that doesn't affect real deployments, and re-enable
accordingly.
References
projects/ores.sql/populate/controller/controller_service_definitions_populate.sql— theenabled = 0hotfix and its inline comment.projects/ores.controller/core/src/service/process_supervisor.cpp— the restart/backoff logic that currently papers over the crash.projects/ores.utility/include/ores.utility/concurrency/retry_strategy.hpp— the exponential-backoff calculator in play.