Star Wars Episode I: The Phantom Menace on the Tab5 (statically recompiled)

.github/workflows/build.yml

@@ -22,7 +22,7 @@ jobs:
       uses: actions/checkout@v4
 
     - name: Checkout the game submodules (without their desktop-only nested submodules)
-      run: git submodule update --init components/jk/OpenJKDF2 components/mgs/mgs_reversing components/mgs/psyz
+      run: git submodule update --init components/jk/OpenJKDF2 components/mgs/mgs_reversing components/mgs/psyz components/tpm/tpm-decomp
 
     - name: Build
       uses: espressif/esp-idf-ci-action@v1

.gitmodules

@@ -9,3 +9,7 @@
 	path = components/mgs/psyz
 	url = https://github.com/esp-cpp/psyz.git
 	branch = esp32p4
+[submodule "components/tpm/tpm-decomp"]
+	path = components/tpm/tpm-decomp
+	url = https://github.com/esp-cpp/tpm-decomp.git
+	branch = main

CMakeLists.txt

@@ -25,6 +25,8 @@ set(SNES_COMPONENTS snes)
 set(GBA_COMPONENTS gba)
 ### Metal Gear Solid (mgs_reversing decompilation on psyz, native) ###
 set(MGS_COMPONENTS mgs)
+### Star Wars Episode I: The Phantom Menace (tpm-decomp on psyz, native) ###
+set(TPM_COMPONENTS tpm)
 
 ### Dark Forces (The Force Engine port, shared with esp-box-emu) ###
 # set(DARK_FORCES_COMPONENTS "dark_forces")
@@ -44,6 +46,9 @@ endif()
 if(MGS_COMPONENTS)
   add_compile_definitions(ENABLE_MGS)
 endif()
+if(TPM_COMPONENTS)
+  add_compile_definitions(ENABLE_TPM)
+endif()
 
 if(DARK_FORCES_COMPONENTS)
   add_compile_definitions(ENABLE_DARK_FORCES)
@@ -55,6 +60,7 @@ set(GAME_COMPONENTS
   ${SNES_COMPONENTS}
   ${GBA_COMPONENTS}
   ${MGS_COMPONENTS}
+  ${TPM_COMPONENTS}
   ${DARK_FORCES_COMPONENTS}
   )
 

README.md

@@ -7,9 +7,11 @@ states, and a `Cart` per game or emulator core, in the same shape as
 [esp-box-emu](https://github.com/esp-cpp/esp-box-emu) so glue can move
 between the two.
 
-Two of the carts are not emulators. They are native ports of PC and
-PlayStation games, running the games' own C code on the two 360 MHz RISC-V
-cores with a software rasterizer.
+Three of the carts are not emulators. They are native ports of PC and
+PlayStation games, running the games' own code as C on the two 360 MHz
+RISC-V cores with a software rasterizer: Jedi Knight and Metal Gear Solid from
+decompiled source, The Phantom Menace from a static recompilation of your own
+copy of the game (see below).
 
 ## Metal Gear Solid, natively
 
@@ -31,6 +33,42 @@ drawn by two rasterizer workers, one per core, each taking a band of rows.
 Save and load states from the pause menu. See
 [components/mgs/README.md](components/mgs/README.md).
 
+## Star Wars Episode I: The Phantom Menace, recompiled
+
+https://github.com/user-attachments/assets/a1001e13-553d-4048-aa23-dba5409316e3
+
+<p>
+<img src="docs/images/tpm-conference.jpg" width="32%"> <img src="docs/images/tpm-corridor.jpg" width="32%"> <img src="docs/images/tpm-pause.jpg" width="32%">
+</p>
+
+The 1999 PlayStation game, USA Rev 1, through
+[tpm-decomp](https://github.com/esp-cpp/tpm-decomp): its static recompiler
+turns the game's executable into C at build time, and its runtime is the
+PlayStation around that C (BIOS, interrupts, DMA, CD-ROM, MDEC, GPU ports)
+on the same psyz rasterizer, GTE and SPU as the Metal Gear Solid port. It
+boots from the disc image on the card through the title, the full-motion
+movies (MDEC video with XA audio at the movies' full 15 fps) and the levels,
+at about 27 fps in game with sound effects, music and speech. Save and
+load states from the pause menu.
+
+**Recompiled, not (yet) decompiled.** Unlike the other two ports, TPM does not
+run from decompiled source. `tpm-decomp/tools/recomp.py` translates every
+MIPS instruction of the executable into C, mechanically. That C is
+effectively Sony's executable in another form, so it is never committed or
+distributed. It is generated in the build directory from **your own** copy
+of `SLUS_008.84` (`CONFIG_TPM_EXE_PATH`), so a playable TPM needs a local
+build. The firmware from CI, built without the executable, carries only the
+cart's load-screen viewer.
+
+The goal is a real, matching decompilation: hand-written C that compiles back
+to the byte-identical executable, as mgs_reversing is for Metal Gear Solid.
+tpm-decomp has started one (the executable is split and rebuilds
+byte-identical, and the first functions are in C), and decompiled functions
+can replace recompiled ones one at a time. Once the decompilation is
+complete, TPM would build from source like the MGS port, with only the game
+data on the card. Contributions are very welcome. See
+[components/tpm/README.md](components/tpm/README.md).
+
 ## Star Wars: Jedi Knight – Dark Forces II
 
 https://github.com/user-attachments/assets/2e6b5173-f53a-4591-8347-96aa53e222f2
@@ -87,6 +125,7 @@ components/
   statistics/       frame timing
   jk/               OpenJKDF2 port: engine submodule + ESP32 platform layer + glue
   mgs/              Metal Gear Solid: mgs_reversing + psyz submodules + the Tab5 platform layer
+  tpm/              The Phantom Menace: tpm-decomp submodule (static recompiler + runtime) + glue
   gba/ genesis/ snes/   the emulator cores and their glue
 docs/PLAN.md        the Jedi Knight plan and history
 ```
@@ -110,16 +149,20 @@ docs/PLAN.md        the Jedi Knight plan and history
   mgs/                        # Metal Gear Solid, extracted from your own US disc
     STAGE.DIR RADIO.DAT FACE.DAT ZMOVIE.STR VOX.DAT DEMO.DAT BRF.DAT
                               # (components/mgs/mgs_reversing/port/extract_disc.py)
+  tpm/                        # The Phantom Menace, USA Rev 1: the .bin of a bin/cue rip
+    Star Wars - Episode I - The Phantom Menace (USA) (Rev 1).bin
   saves/                      # save states and their screenshots
 ```
 
 Example `metadata.csv` (one line per game: rom path, boxart, name; the
 platform comes from the path: `.md`/`.gen` Genesis, `.sfc`/`.smc` SNES,
-`.gba` Game Boy Advance, `jk.cd` Jedi Knight, `STAGE.DIR` Metal Gear Solid):
+`.gba` Game Boy Advance, `jk.cd` Jedi Knight, `STAGE.DIR` Metal Gear Solid,
+`SLUS_008.84` The Phantom Menace):
 
 ```
 jk/jk.cd, boxart/jk.jpg, Star Wars: Jedi Knight - Dark Forces II
 mgs/STAGE.DIR, boxart/mgs.jpg, Metal Gear Solid
+tpm/SLUS_008.84, boxart/tpm.jpg, Star Wars Episode I: The Phantom Menace
 genesis/sonic.md, boxart/sonic.jpg, Sonic the Hedgehog
 snes/smw.sfc, boxart/smw.jpg, Super Mario World
 gba/mk.gba, boxart/mk.jpg, Mario Kart: Super Circuit
@@ -136,6 +179,7 @@ Requires ESP-IDF v6.x.
 git clone --recurse-submodules=components/jk/OpenJKDF2 \
           --recurse-submodules=components/mgs/mgs_reversing \
           --recurse-submodules=components/mgs/psyz \
+          --recurse-submodules=components/tpm/tpm-decomp \
           https://github.com/esp-cpp/tab5-emu
 cd tab5-emu
 idf.py set-target esp32p4

components/gui/include/gui.hpp

@@ -5,6 +5,7 @@
 #include <memory>
 #include <mutex>
 #include <optional>
+#include <string_view>
 #include <vector>
 
 #include "lvgl.h"
@@ -47,6 +48,9 @@ class Gui {
 
   void update_rom_list();
 
+  /// Select the first game whose name or rom path contains `text`
+  bool select_rom_matching(std::string_view text);
+
 protected:
   void init_ui();
   void deinit_ui();

components/gui/src/gui.cpp

@@ -240,6 +240,18 @@ void Gui::update_rom_list() {
   }
 }
 
+bool Gui::select_rom_matching(std::string_view text) {
+  std::lock_guard<std::recursive_mutex> lk(mutex_);
+  for (size_t i = 0; i < rom_infos_.size(); i++) {
+    const auto &rom = rom_infos_[i];
+    if (rom.name.find(text) != std::string::npos || rom.rom_path.find(text) != std::string::npos) {
+      select_rom(i);
+      return true;
+    }
+  }
+  return false;
+}
+
 void Gui::select_rom(int index) {
   std::lock_guard<std::recursive_mutex> lk(mutex_);
   if (index < 0 || index >= (int)rom_infos_.size()) {

components/mgs/src/mgs_platform.cpp

@@ -40,6 +40,7 @@ void mgs_platform_init(const char *data_dir) { g_sd_root = data_dir ? data_dir :
 
 extern "C" {
 void Psyz_SpuInit(void);
+extern int psyz_spu_by_voice; // psyz_spu.c: 1 mixes a block a voice at a time
 void Psyz_SpuPullSamples(short *out, int num_frames);
 extern volatile unsigned psyz_spu_irq_raised, psyz_spu_voices_active, psyz_spu_irq_caught_up; // psyz_spu.c
 extern volatile unsigned mgs_spu_irq_delivered;                        // port/esp32_vblank.c
@@ -55,6 +56,12 @@ std::atomic<bool> g_audio_stop{false};
 std::atomic<bool> g_audio_done{false};
 
 void audio_task(void *) {
+  // MGS clocks its sound driver off the SPU interrupt (voice 23 loops a blank
+  // sample, the IRQ address toggled ahead of it): mixed a voice at a time,
+  // the beats came irregularly (a third of them re-armed behind the voice
+  // and caught up, against a tenth sample by sample) and the opening's
+  // streamed audio was choppier. Sample by sample costs ~2.5x the mixing.
+  psyz_spu_by_voice = 0;
   auto &emu = Tab5Emu::get();
   static int16_t buf[kAudioFrames * 2];
   int64_t last_report = esp_timer_get_time();

components/rom_info/include/rom_info.hpp

@@ -7,7 +7,7 @@
 #include "format.hpp"
 
 
-enum class Emulator { UNKNOWN, JEDI_KNIGHT, DARK_FORCES, SEGA_GENESIS, SEGA_MEGA_DRIVE, SNES, GBA, MGS };
+enum class Emulator { UNKNOWN, JEDI_KNIGHT, DARK_FORCES, SEGA_GENESIS, SEGA_MEGA_DRIVE, SNES, GBA, MGS, TPM };
 
 struct RomInfo {
   std::string name;
@@ -45,6 +45,8 @@ template <> struct fmt::formatter<Emulator> {
       return fmt::format_to(ctx.out(), "DARK_FORCES");
     case Emulator::MGS:
       return fmt::format_to(ctx.out(), "MGS");
+    case Emulator::TPM:
+      return fmt::format_to(ctx.out(), "TPM");
     default:
       return fmt::format_to(ctx.out(), "UNKNOWN");
     }

components/rom_info/src/rom_info.cpp

@@ -74,6 +74,10 @@ std::vector<RomInfo> parse_metadata(const std::string &metadata_path) {
     } else if (endsWith(lp, "stage.dir") || endsWith(lp, ".mgs")) { // metal gear solid
 #if defined(ENABLE_MGS)
       platform = Emulator::MGS;
+#endif
+    } else if (endsWith(lp, "slus_008.84")) { // star wars episode i: the phantom menace
+#if defined(ENABLE_TPM)
+      platform = Emulator::TPM;
 #endif
     }
     if (platform != Emulator::UNKNOWN) {

components/tpm/CMakeLists.txt

@@ -0,0 +1,147 @@
+# Star Wars Episode I: The Phantom Menace (PSX), natively: tpm-decomp
+# (https://github.com/esp-cpp/tpm-decomp, submodule tpm-decomp/).
+#
+# With CONFIG_TPM_ENGINE, the game's executable is statically recompiled to C
+# at configure time (tpm-decomp/tools/recomp.py) and built with tpm-decomp's
+# runtime (BIOS, interrupts, DMA, CD-ROM, GPU ports) on psyz's software GPU,
+# GTE and SPU. The executable is the user's own: CONFIG_TPM_EXE_PATH. Without
+# it the component builds only the platform half (src/), which shows the
+# game's load screens from the SD card.
+
+set(TPM_DECOMP ${CMAKE_CURRENT_LIST_DIR}/tpm-decomp)
+# psyz: the copy components/mgs already carries (same revision tpm-decomp pins)
+set(TPM_PSYZ ${CMAKE_CURRENT_LIST_DIR}/../mgs/psyz)
+
+set(TPM_SRCS "src/tpm.cpp" "src/tpm_files.cpp")
+set(TPM_ENGINE_READY OFF)
+set(TPM_GEN ${CMAKE_CURRENT_BINARY_DIR}/recomp)
+
+if(CONFIG_TPM_ENGINE)
+  set(exe "${CONFIG_TPM_EXE_PATH}")
+  if(NOT IS_ABSOLUTE "${exe}")
+    set(exe "${CMAKE_SOURCE_DIR}/${exe}")
+  endif()
+  if(NOT EXISTS "${exe}")
+    message(WARNING "tpm: ${exe} not found (CONFIG_TPM_EXE_PATH): building without the game; "
+                    "the cart will show the load screens only")
+  elseif(NOT EXISTS "${TPM_PSYZ}/psyz/src/platform/soft_raster.inc.c")
+    message(WARNING "tpm: psyz missing at ${TPM_PSYZ} (git submodule update --init components/mgs/psyz)")
+  else()
+    idf_build_get_property(python PYTHON)
+    if(NOT python)
+      set(python python3)
+    endif()
+    # Re-run when the executable or the recompiler changes
+    # the access sites that go to the scratchpad check it first
+    # the movie player overlay comes from the disc image, when there is one
+    set(disc_arg)
+    set(disc "${CONFIG_TPM_DISC_PATH}")
+    if(disc)
+      if(NOT IS_ABSOLUTE "${disc}")
+        set(disc "${CMAKE_SOURCE_DIR}/${disc}")
+      endif()
+      if(EXISTS "${disc}")
+        set(disc_arg --disc "${disc}")
+        set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${disc}")
+      else()
+        message(WARNING "tpm: ${disc} not found (CONFIG_TPM_DISC_PATH): no movies")
+      endif()
+    endif()
+    set(sites ${TPM_DECOMP}/config/scratch_sites.us.txt)
+    set(sites_arg)
+    if(EXISTS "${sites}")
+      set(sites_arg --scratch-sites "${sites}")
+    endif()
+    set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS
+      "${exe}" "${TPM_DECOMP}/tools/recomp.py" "${TPM_DECOMP}/config/recomp_entries.us.txt"
+      "${TPM_DECOMP}/recomp/hle.us.txt" "${sites}")
+    execute_process(
+      COMMAND ${python} ${TPM_DECOMP}/tools/recomp.py "${exe}" ${TPM_GEN} --files 16 ${sites_arg} ${disc_arg}
+      RESULT_VARIABLE rc OUTPUT_VARIABLE out ERROR_VARIABLE err)
+    if(rc EQUAL 0)
+      set(TPM_ENGINE_READY ON)
+      string(REGEX MATCH "recomp: [0-9]+ functions[^\n]*" summary "${err}")
+      message(STATUS "tpm: ${summary}")
+    else()
+      message(WARNING "tpm: recompiling ${exe} failed:\n${err}")
+    endif()
+  endif()
+endif()
+
+if(TPM_ENGINE_READY)
+  list(APPEND TPM_SRCS "src/tpm_engine.cpp")
+endif()
+
+idf_component_register(
+  SRCS ${TPM_SRCS}
+  INCLUDE_DIRS "include"
+  PRIV_INCLUDE_DIRS "src" "${TPM_DECOMP}/recomp/runtime"
+  REQUIRES tab5-emu
+  PRIV_REQUIRES esp_timer heap statistics shared_memory esp_app_format
+)
+
+if(TPM_ENGINE_READY)
+  target_compile_definitions(${COMPONENT_LIB} PRIVATE TPM_HAVE_ENGINE)
+
+  # ---- the game + runtime + the psyz pieces it uses, as one library ----
+  set(rt ${TPM_DECOMP}/recomp/runtime)
+  file(GLOB rt_srcs ${rt}/*.c)
+  file(GLOB gen_srcs ${TPM_GEN}/recomp_*.c)
+  set(psyz_srcs ${TPM_PSYZ}/psyz/src/psyz/psyz_spu.c) # (the GTE is the runtime's own: rt_gte.c)
+  add_library(tpm_rt STATIC ${gen_srcs} ${rt_srcs} ${psyz_srcs})
+  idf_build_get_property(config_dir CONFIG_DIR)
+  target_include_directories(tpm_rt PRIVATE
+    ${rt} ${TPM_GEN} ${TPM_PSYZ}/psyz/include ${TPM_PSYZ}/psyz/src
+    ${TPM_PSYZ}/decomp/src/libspu ${config_dir})
+  # IDF's own flags (the arch flags come with the toolchain file)
+  idf_build_get_property(idf_opts COMPILE_OPTIONS GENERATOR_EXPRESSION)
+  idf_build_get_property(idf_c_opts C_COMPILE_OPTIONS GENERATOR_EXPRESSION)
+  idf_build_get_property(idf_defs COMPILE_DEFINITIONS GENERATOR_EXPRESSION)
+  target_compile_options(tpm_rt PRIVATE ${idf_opts} ${idf_c_opts})
+  target_compile_definitions(tpm_rt PRIVATE ${idf_defs} ESP_PLATFORM)
+  target_link_libraries(tpm_rt PRIVATE idf::heap idf::esp_hw_support idf::esp_common idf::esp_rom
+                                       idf::freertos idf::log)
+  #   -O2                   the game's code: speed over the IDF's size setting
+  #   -fno-strict-aliasing  guest memory is read through any type
+  #   -msmall-data-limit=0  nothing in PSRAM may be addressed gp-relative
+  #   -ffixed-s11           s11 holds the guest RAM's address for the game's code
+  #                         (RT_RAM_REG): nothing in the bundle may use it
+  target_compile_options(tpm_rt PRIVATE -O2 -w -fno-strict-aliasing -msmall-data-limit=0
+                                        -Wno-error -ffixed-s11)
+  target_compile_definitions(tpm_rt PRIVATE RT_RAM_REG=s11)
+  # the generated functions are huge and short of registers: scheduling
+  # that minds the pressure (-1%; without the scheduling pass, +2%), and s0
+  # for the code instead of a frame pointer (-0.5%)
+  set_property(SOURCE ${gen_srcs} APPEND PROPERTY COMPILE_OPTIONS "-fsched-pressure;-fomit-frame-pointer")
+  # the rasterizer's inner loops decide the frame rate; they run from
+  # PSRAM-XIP, not IRAM (the runtime's default, as in the MGS port: internal
+  # RAM is what the boot-time DMA reserve and the HAL need)
+  set_property(SOURCE ${rt}/rt_gpu.c APPEND PROPERTY COMPILE_OPTIONS "-O3;-fomit-frame-pointer")
+  # ...as are the SPU's per-sample loop and the GTE's commands, on the game's
+  # thread
+  set_property(SOURCE ${TPM_PSYZ}/psyz/src/psyz/psyz_spu.c ${rt}/rt_gte.c APPEND PROPERTY
+               COMPILE_OPTIONS "-O3;-fomit-frame-pointer")
+
+  # ---- hide it: components/mgs links its own psyz, with the same names ----
+  # One relocatable object with only the runtime's API left global.
+  set(bundle ${CMAKE_CURRENT_BINARY_DIR}/tpm_rt_bundle.o)
+  # tpm_rt.ld gathers the statics: .bss to PSRAM, both bracketed for the
+  # reset between launches (-d: COMMON symbols get their space here)
+  add_custom_command(OUTPUT ${bundle}
+    COMMAND ${CMAKE_C_COMPILER} -nostdlib -r -Wl,-d -Wl,-T,${CMAKE_CURRENT_LIST_DIR}/tpm_rt.ld
+            -Wl,--whole-archive $<TARGET_FILE:tpm_rt> -Wl,--no-whole-archive -o ${bundle}.r.o
+    COMMAND ${CMAKE_OBJCOPY} --keep-global-symbols=${CMAKE_CURRENT_LIST_DIR}/tpm_rt.keep
+            ${bundle}.r.o ${bundle}
+    DEPENDS tpm_rt ${CMAKE_CURRENT_LIST_DIR}/tpm_rt.keep ${CMAKE_CURRENT_LIST_DIR}/tpm_rt.ld
+    VERBATIM)
+  add_custom_target(tpm_rt_bundle DEPENDS ${bundle})
+  add_dependencies(${COMPONENT_LIB} tpm_rt_bundle)
+  target_link_libraries(${COMPONENT_LIB} PRIVATE ${bundle})
+
+  # bring-up: a GPU recording to replay as a benchmark (TPM_GPU_BENCH_FILE)
+  if(NOT "${CONFIG_TPM_GPU_BENCH_FILE}" STREQUAL "")
+    configure_file("${CONFIG_TPM_GPU_BENCH_FILE}" ${CMAKE_CURRENT_BINARY_DIR}/gpubench.bin COPYONLY)
+    target_add_binary_data(${COMPONENT_LIB} ${CMAKE_CURRENT_BINARY_DIR}/gpubench.bin BINARY)
+    target_compile_definitions(${COMPONENT_LIB} PRIVATE TPM_GPU_BENCH=1)
+  endif()
+endif()