Beware foreign memory in lisp images

Tagged as lisp

Written on 2019-10-21 17:50:00

If you're developing a game in Common Lisp, two things are probably true:

• you're using CFFI to call into a C library, like SDL2.
• you're building a lisp image to ship your game.

Taken together, you're open to a very subtle bug.

Foreign memory isn't consistent across lisp image builds. This means the bits allocated during building won't map to the same bits when the lisp image is started.

This has the effect of the code behaving normally when run from the REPL, but buggy when run from the lisp image.

A quick example. We'll use this code:

(in-package :cl-user)
(ql:quickload :cffi)

(defvar *foreign-array*
  (let ((foreign-pointer (cffi:foreign-alloc :int :count 10)))
    (loop :for i :from 0 :below 10 :do
         (setf (cffi:mem-aref foreign-pointer :int i) i))
    foreign-pointer))

(defun print-foreign-array (foreign-pointer array-length)
  (format t "printing foreign array:~%")
  (loop :for i :from 0 :below array-length :do
       (format t "  ~A: ~A~%" i (cffi:mem-aref foreign-pointer :int i))))

(defun run-example ()
  (print-foreign-array *foreign-array* 10))

(defun make-lisp-image ()
  (save-lisp-and-die
   "example"
   :purify T
   :toplevel #'run-example
   :executable t
   :save-runtime-options t))

Running our example from the REPL works as expected.

CL-USER> (run-example)
printing foreign array:
  0: 0
  1: 1
  2: 2
  3: 3
  4: 4
  5: 5
  6: 6
  7: 7
  8: 8
  9: 9

The image, on the other hand, yields very different results!

~/prog/recurseblog $ sbcl --load foreign-memory.lisp --eval "(make-lisp-image)"
~/prog/recurseblog $ chmod +x example && ./example
printing foreign array:
CORRUPTION WARNING in SBCL pid 12734(tid 0x7fc06695ab80):
Memory fault at 0x253f010 (pc=0x52189073, fp=0x7fc0591d7d30, sp=0x7fc0591d7cf0) tid 0x7fc06695ab80
The integrity of this image is possibly compromised.
Continuing with fingers crossed.

debugger invoked on a SB-SYS:MEMORY-FAULT-ERROR in thread
#<THREAD "main thread" RUNNING {10005404C3}>:
  Unhandled memory fault at #x253F010.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit from the current thread.

(PRINT-FOREIGN-ARRAY #.(SB-SYS:INT-SAP #X0253F010) 10)
   source: (CFFI:MEM-AREF FOREIGN-POINTER :INT I)
0]

Keep in mind, a crash is the best case scenario! If you're unlucky your program will continue to run with corrupted data.

The solution is to free all foreign memory before building the saved image, then re-allocating on image startup.

(defun pre-image-save ()
  (cffi:foreign-array-free *foreign-array*)
  (setf *foreign-array* nil))

(defun post-image-startup ()
  (setf *foreign-array*
        (let ((foreign-pointer (cffi:foreign-alloc :int :count 10)))
          (loop :for i :from 0 :below 10 :do
               (setf (cffi:mem-aref foreign-pointer :int i) i))
          foreign-pointer)))

(defun make-lisp-image ()
  (pre-image-save)
  (save-lisp-and-die
   "example"
   :purify T
   :toplevel #'(lambda ()
                 (post-image-startup)
                 (run-example))
   :executable t
   :save-runtime-options t))

Now everything works as expected

~/prog/recurseblog $ ros --load foreign-memory.lisp --eval "(make-lisp-image)"
~/prog/recurseblog $ ./example
printing foreign array:
  0: 0
  1: 1
  2: 2
  3: 3
  4: 4
  5: 5
  6: 6
  7: 7
  8: 8
  9: 9

• ← Previous
• Next →

---

If you'd like to follow my indie game, Syn, please consider Wishlisting on Steam or Subscribing to Syn's email list.