From 6c91ac9cec29ebd36193a78e33e732f55e6e5e6f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kacper=20Michaj=C5=82ow?= <kasper93@gmail.com>
Date: Mon, 7 Apr 2025 21:47:27 +0200
Subject: [PATCH] Add -fno-strict-float-cast-overflow complation flag
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

LuaJIT checks whether a Lua number can be represented as an integer by
casting the number (`double`) to an `int32_t` and comparing the result
to the original. While this approach works in practice, it is
technically UB according to the C standard. Because it's UB, compilers
are free to make optimizations that may not preserve the original intent
of the code. Clang/LLVM, in particular, takes advantage of this.

Consider the following code:

```c
int32_t k = lj_num2int(n);
if (n == (lua_Number)k)
  ...
```

Clang short-circuit this to do basically

```c
if (n == trunc(n))
  ...
```

It can do this because it assumes that `n` is either representable as an
`int32_t` or the behavior is undefined, which allows it to compare the
truncated `n` directly without needing to convert `k` back to a
`double`.

We could implement a C-compliant saturating cast from floating point to
integer ourselves, but that awkward, and more useless code, especially
since most FPUs already handle such conversions natively with
saturation.

Alternatively, we could use intrinsics or inline asm to ensure the
correct saturating conversion is used.

But there's a simpler option: we can ask the compiler to handle this
correctly for us.

This commit adds the `-fno-strict-float-cast-overflow` compilation flag, which:
> With the ‘no-strict’ option, Clang will saturate towards the smallest
and largest representable integer values instead. NaNs will be converted
to zero.

GCC does not currently exploit this UB as aggressively, so it's not an
issue there. MSVC also appears unaffected, though it provides built-in
functions for saturating conversions, if someone wants to use them
directly
https://learn.microsoft.com/cpp/intrinsics/saturation-conversion-functions

This commit fixes assumptions that LuaJIT makes in the code.

Fixes: #1351
---
 src/Makefile | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/Makefile b/src/Makefile
index 4a56d1e8..c83abfa0 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -302,6 +302,9 @@ endif
 ifneq (,$(INSTALL_LJLIBD))
   TARGET_XCFLAGS+= -DLUA_LJDIR=\"$(INSTALL_LJLIBD)\"
 endif
+ifeq (,$(shell $(TARGET_CC) -o /dev/null -c -x c /dev/null -fno-strict-float-cast-overflow 2>/dev/null || echo 1))
+  TARGET_XCFLAGS+= -fno-strict-float-cast-overflow
+endif
 
 ##############################################################################
 # Target system detection.